Object discovery and exploration in video content

ABSTRACT

A real-time video exploration (RVE) system that allows users to pause, step into, move through, and explore 2D or 3D modeled worlds of scenes in a video. The RVE system may allow users to discover, select, explore, and manipulate objects within the modeled worlds used to generate video content. The RVE system may implement methods that allow users to view and explore in more detail the features, components, and/or accessories of selected objects that are being manipulated and explored. The RVE system may also implement methods that allow users to interact with interfaces of selected objects or interfaces of components of selected objects.

This application claims benefit of priority of U.S. ProvisionalApplication Ser. No. 61/951,495 entitled “OBJECT DISCOVERY ANDEXPLORATION IN VIDEO CONTENT” filed Mar. 11, 2014, the content of whichis incorporated by reference herein in its entirety.

BACKGROUND

Much video produced today, including but not limited to movies, shorts,cartoons, commercials, and television and cable programs, is at leastpartially generated using two-dimensional (2D) or three-dimensional (3D)computer graphics techniques. For example, modern animated movies aretypically generated using various 3D computer graphics techniques asimplemented by various 3D graphics applications to generate 3Drepresentations or models of scenes, and then applying 3D renderingtechniques to render two-dimensional (2D) representations of the 3Dscenes. As another example, scenes in some video such as movies may begenerated by filming live actor(s) using green- or blue-screentechnology, and filling in the background and/or adding other content oreffects using one or more 3D computer graphics techniques.

Generating a scene using computer graphics techniques may, for example,involve generating a background for the scene, generating one or moreobjects for the scene, combining the background and objects(s) into arepresentation or model of the scene, and applying rendering techniquesto render a representation of the model of the scene as output. Eachobject in a scene may be generated according to an object model thatincludes but is not limited to an object frame or shape (e.g., a wireframe), surface texture(s), and color(s). Rendering of a scene mayinclude applying global operations or effects to the scene such asillumination, reflection, shadows, and simulated effects such as rain,fire, smoke, dust, and fog, and may also include applying othertechniques such as animation techniques for the object(s) in the scene.Rendering typically generates as output sequences of 2D video frames forthe scenes, and the video frame sequences may be joined, merged, andedited as necessary to generate final video output, for example a movie.

In video production, for example in movie production that uses 2D or 3Dtechniques as described above, a director (or other entity) selects aviewpoint or perspective for each scene, and the final output is a video(e.g., a movie) that presents a 2D representation of the environmentsthat were generated and used to render the video, with each frame ofeach scene shown from a pre-selected perspective. Thus, a consumer ofthe video (e.g., an animated movie) views the scenes in the movie fromperspectives that were pre-selected by the director, and all consumersview the movie from the same perspectives.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a high-level illustration of a real-time video exploration(RVE) system, according to at least some embodiments.

FIG. 1B illustrates an example real-time video exploration (RVE) systemand environment in which users can explore modeled worlds rendered inreal-time during playback of pre-recorded video, according to at leastsome embodiments.

FIG. 1C illustrates an example RVE client system, according to at leastsome embodiments.

FIG. 2 is a flowchart of a method for exploring modeled worlds inreal-time during playback of pre-recorded video, according to at leastsome embodiments.

FIG. 3 is a flowchart of a method for interacting with objects andrendering new video content of the manipulated objects while exploring avideo being played back, according to at least some embodiments.

FIG. 4 is a flowchart of a method for modifying and ordering objectswhile exploring a video being played back, according to at least someembodiments.

FIG. 5 is a flowchart of a method for rendering and storing new videocontent during playback of pre-recorded video, according to at leastsome embodiments.

FIG. 6A is a high-level illustration of a real-time video exploration(RVE) system that enables the generation of new video from pre-recordedvideo, according to at least some embodiments.

FIGS. 6B and 6C illustrate example real-time video exploration (RVE)systems and environments in which users can render and store new videocontent during playback of a pre-recorded video, according to at leastsome embodiments.

FIGS. 7A through 7C graphically illustrate exploring a rendered 3D modelof a scene while a pre-recorded video is paused, according to at leastsome embodiments.

FIG. 8A graphically illustrates selecting an object and obtaininginformation about the object while exploring a rendered model of ascene, according to at least some embodiments.

FIG. 8B graphically illustrates manipulating a rendering of a selectedobject while exploring a rendered model of a scene, according to atleast some embodiments.

FIG. 9 graphically illustrates interacting with components of a selectedobject while exploring a rendered model of a scene, according to atleast some embodiments.

FIG. 10 illustrates an example RVE system and environment in which aclient device uses an external control device to explore the videocontent, according to at least some embodiments.

FIG. 11 illustrates an example RVE system and environment in which aclient device uses a “second screen” to explore the video content,according to at least some embodiments.

FIG. 12 illustrates an example RVE system and environment in whichobjects in a pre-recorded video can be modified, new video contentincluding the modified objects can be generated and streamed, andobjects from the video can optionally be ordered, according to at leastsome embodiments.

FIG. 13 illustrates an example network-based RVE environment, accordingto at least some embodiments.

FIG. 14 illustrates an example network-based environment in which astreaming service is used to stream rendered video to clients, accordingto at least some embodiments.

FIG. 15 is a diagram illustrating an example provider networkenvironment in which embodiments as described herein may be implemented.

FIG. 16 is a block diagram illustrating an example computer system thatmay be used in some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include”, “including”, and“includes” mean including, but not limited to.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for generating, presenting,and exploring two-dimensional (2D) or three-dimensional (3D) modeledworlds from within pre-rendered video are described. Video, includingbut not limited to movies, may be produced using 2D or 3D computergraphics techniques to generate 2D or 3D modeled worlds for scenes andrender representations of the modeled worlds from selected cameraviewpoints as output. In video production, scene content (e.g., objects,textures, colors, backgrounds, etc.) is determined for each scene, acamera viewpoint or perspective is pre-selected for each scene, thescenes (each representing a 2D or 3D world) are generated and renderedaccording to computer graphics techniques, and the final rendered outputvideo (e.g., a movie) includes a representation of the modeled worlds,with each frame of each scene rendered and shown from a fixed,pre-selected camera viewpoint and angle, and with fixed, predeterminedcontent. Thus, conventionally, a consumer of pre-rendered video (e.g., amovie) views the scenes in the movie from pre-selected camera viewpointsand angles, and with pre-determined content.

Large amounts of 2D or 3D graphics data may be used in generating andrendering scenes for video (e.g., for movies) according to computergraphics techniques. Note that this graphics data may be used in 2D or3D rendering of video content according to different productiontechniques, for example in producing fully rendered, animated videocontent according to computer graphics techniques as well as inproducing partially rendered video content that involves filming liveaction using green- or blue-screen technology and filling in thebackground and/or adding other content or effects using one or morecomputer graphics techniques. For a given scene, this graphics data mayinclude, but is not limited to, 2D or 3D object model data such asobject frames or shapes (e.g., wire frames), wraps for the frames,surface textures and patterns, colors, animation models, and so on, thatis used to generate models of objects for the scene; general sceneinformation such as surfaces, vanishing points, textures, colors,lighting sources, and so on; information for global operations oreffects in the scenes such as illumination, reflection, shadows, andsimulated effects such as rain, fire, smoke, dust, and fog; and ingeneral any information or data that may be used in generating a modeledworld for the scene and in rendering 2D representations of the world(e.g., video frames) as video output. This graphics data used ingenerating videos (e.g., movies) includes rich 2D or 3D content that isnot presented to the viewer in conventional video, as the viewer viewsthe scenes in the video rendered from perspectives that werepre-selected by the director, and all viewers of the video view thescenes from the same perspectives. However, this graphics data may beavailable or may be made available, and if not available at least somegraphics data may be generated from the original video, for exampleusing various 2D-to-3D modeling techniques.

Embodiments of real-time video exploration (RVE) methods and systems aredescribed that may leverage this 2D or 3D graphics data to enableinteractive exploration of 2D or 3D modeled worlds from scenes inpre-rendered, pre-recorded video by generating and rendering new videocontent in real time at least in part from the 2D or 3D graphics data.Embodiments of the RVE methods and systems are generally describedherein with respect to interactive exploration of 3D modeled worlds.However, embodiments may also be applied in generating and rendering 2Dmodels and objects for video using 2D graphics techniques to enableinteractive exploration of 2D modeled worlds.

FIG. 1A is a high-level illustration of a real-time video exploration(RVE) system 10, according to at least some embodiments. Embodiments ofan RVE system 10 may, for example, allow a video consumer (also referredto herein as a user or viewer), via an RVE client 30, to “step into” ascene in a video (e.g., a movie) to explore the rest of the 3D modeledworld “behind the scenes” via a user-controlled, free-roaming “camera”that allows the user to change viewing positions and angles in the 3Dmodeled world.

In at least some embodiments, the RVE system 10 may play back video fromone or more sources 20 to one or more RVE clients 30, receive userinput/interactions within scenes being explored from respective RVEclients 30, responsively generate or update 3D models from graphics dataobtained from one or more sources 20 in response to the userinput/interactions exploring the scenes, render new video content of thescenes at least in part from the 3D models, and deliver the newlyrendered video content (and audio, if present) to the respective RVEclients 30 as RVE video. Thus, rather than just viewing a pre-renderedscene in a movie from a perspective that was pre-selected by a director,a user may step into and explore the scene from different angles, wanderaround the scene at will within the scope of the 3D modeled world, anddiscover hidden objects and/or parts of the scene that are not visiblein the original video as recorded. The RVE video that is output to theclient(s) 30 by RVE system 10 is a video stream that has been processedand rendered according to two inputs, one input being the user'sexploratory inputs, the second input being the recorded video and/orgraphics data obtained from source(s) 20. In at least some embodiments,RVE system 10 may provide one or more application programming interfaces(APIs) for receiving input from and sending output to RVE client(s) 30.

Since exploring and rendering a 3D world is computationally expensive,at least some embodiments of an RVE system 10 may leverage network-basedcomputation resources and services (e.g., a streaming service) toreceive user input/interactions within a scene being explored from anRVE client 30 on a client device, responsively generate or update a 3Dmodel from the 3D data in response to the user input/interactions,render new video content of the scene from the 3D model, and deliver thenewly rendered video content (and in some cases also audio) as a videostream to the client device in real-time or near-real-time and with lowlatency. The computational power available through the network-basedcomputation resources, as well as the video and audio streamingcapabilities provided through a streaming protocol, allows the RVEsystem 10 to provide low-latency responses to the user's interactionswith the 3D world as viewed on the respective client device, thusproviding a responsive and interactive exploratory experience to theuser. FIG. 13 illustrates an example RVE system and environment in whichnetwork-based computation resources are leveraged to provide real-time,low-latency rendering and streaming of video content, according to atleast some embodiments. FIG. 14 illustrates an example network-basedenvironment in which a streaming service is used to stream renderedvideo to clients, according to at least some embodiments. FIG. 15illustrates an example provider network environment in which embodimentsof an RVE system as described herein may be implemented. FIG. 16 is ablock diagram illustrating an example computer system that may be usedin some embodiments.

In addition to allowing users to pause, step into, move through, andexplore the 3D modeled worlds of scenes in a video, at least someembodiments of an RVE system 10 may also allow users to modify thescenes, for example by adding, removing, or modifying various graphicseffects such as lens effects (e.g., fisheye, zoom, filter, etc.),lighting effects (e.g., illumination, reflection, shadows, etc.), coloreffects (color palette, color saturation, etc.), or various simulatedeffects (e.g., rain, fire, smoke, dust, fog, etc.) to the scenes.

In addition to allowing users to pause, step into, move through,explore, and even modify the 3D modeled worlds of scenes in a video, atleast some embodiments of an RVE system 10 may also allow users todiscover, select, explore, and manipulate objects within the 3D modeledworlds used to generate video content. At least some embodiments of anRVE system 10 may implement methods that allow users to view and explorein more detail the features, components, and/or accessories of selectedobjects that are being manipulated and explored. At least someembodiments of an RVE system 10 may implement methods that allow usersto interact with interfaces of selected objects or interfaces ofcomponents of selected objects.

In addition to allowing users to explore scenes and manipulate objectswithin scenes, at least some embodiments of an RVE system 10 may allowusers to interact with selected objects to customize or accessorize theobjects. For example, a viewer can manipulate or interact with aselected object to add or remove accessories, customize the object(change color, texture, etc.), or otherwise modify the object accordingto the user's preferences or desires. In at least some embodiments, theRVE system 10 may provide an interface via which the user can obtainadditional information for the object, customize and/or accessorize anobject if and as desired, be given a price or price(s) for the object ascustomized/accessorized, and order or purchase a physical version of theobject as specified if desired.

In at least some embodiments, a user may order, purchase, or obtain avirtual representation of the object instead of or in addition to aphysical version of the object, if desired. A virtual representation maybe any digital representation of a physical product, item, or object. Avirtual representation may be any type of digital representation fromstatic or animated 2D or 3D digital images or graphics to complex 2D or3D models (e.g., computer-aided design (CAD) models, computer-generatedimagery (CGI) models, etc.) that may, for example, be instantiated,rendered, and in some cases animated and manipulated within virtualuniverses by physics engines.

At least some embodiments of an RVE system 10 may allow a user to createand record their own customized version of a video such as a movie,and/or to stream or broadcast a customized version of a video to one ormore destinations in real time. Using embodiments, new versions ofvideos or portions of videos may be generated and may, for example, bestored or recorded to local or remote storage, shown to or shared withfriends, or may be otherwise recorded, stored, shared, streamed,broadcast, or distributed assuming the acquisition of appropriate rightsand permissions to share, distribute, or broadcast the new videocontent.

At least some embodiments of an RVE system 10 may leverage network-basedcomputation resources and services to allow multiple users tosimultaneously receive, explore, manipulate, and/or customize apre-recorded video via clients 30. The RVE system 10 may, for example,broadcast a video stream to multiple clients 30, and users correspondingto the clients 30 may each explore, manipulate, and/or customize thevideo as desired. Thus, at any given time, two or more users may besimultaneously exploring a given scene of a video being played back inreal time, or may be simultaneously watching the scene from differentperspectives or with different customizations, with the RVE system 10interactively generating, rendering, and streaming new video to clients30 corresponding to the users according to the users' particularinteractions with the video. Note that the video being played back tothe clients 30 may be pre-recorded video or may be new video generatedby a user via one of the clients 30 and broadcast “live” to one or moreothers of the clients 30 via the RVE system 10.

At least some embodiments of an RVE system 10 may leverage network-basedcomputation resources and services, available 3D model data, andavailable viewer information to dynamically personalize content of, oradd personalized content to, video for particular viewers. Usingembodiments, video (e.g., a movie) can be pre-recorded, and when playedback to viewers, at least some objects in at least some of the scenes ofthe pre-recorded video may be replaced with objects targeted atparticular viewers according to profiles of the viewers. Since the videois being rendered and streamed to different viewers in real-time by thenetwork-based computation resources and services, any given scene of avideo being streamed to the viewers may be modified and viewed in manydifferent ways by different viewers based on the particular viewers'profiles.

FIGS. 1B through 12 illustrate embodiments and operations of an RVEsystem 10 and RVE environment in more detail. FIG. 13 illustrates anexample provider network environment in which network-based computationand storage resources are leveraged to implement components or modulesof an RVE system 10. FIG. 14 illustrates an example network-based RVEenvironment in which a streaming service provides an interface forstreaming rendered video to clients. FIG. 15 illustrates an exampleprovider network environment in which embodiments of an RVE system 10may be implemented. FIG. 16 illustrates an example computer system thatmay be used in embodiments of an RVE system 10. While embodiments of theRVE system 10 are generally described as generating 3D models of scenesand objects and rendering video from the 3D models of scenes and 3Dobjects using 3D graphics techniques, embodiments may also be applied ingenerating and rendering 2D models and objects for video using 2Dgraphics techniques.

Real-time Exploration of Video Content

At least some embodiments of a real-time video exploration (RVE) system10 may implement methods that allow users to pause, step into, movethrough, and explore the 3D modeled worlds used to generate videocontent (e.g., scenes in movies or other video) during playback of apreviously recorded video. Leveraging network-based computationresources and services and utilizing the rich 3D content and data thatwas used to generate and render the original, previously rendered andrecorded video, the RVE system 10 may allow a viewer or viewers of avideo, for example a movie, to pause and “step into” a 3D rendered scenefrom the video, move through the scene to change their point of view,and to thus view and explore the scene and objects in the scene fromdifferent angles than the pre-determined angles used in generating theoriginal video.

FIG. 1B illustrates an example real-time video exploration (RVE) system100 in an RVE environment in which users can explore 3D modeled worldsrendered in real-time during playback of pre-recorded video, accordingto at least some embodiments. FIG. 13 illustrates an example providernetwork environment in which network-based computation and storageresources are leveraged to implement components or modules of RVE system100. FIG. 14 illustrates an example network-based RVE environment inwhich a streaming service provides an interface for streaming renderedvideo to clients. FIG. 15 illustrates an example provider networkenvironment in which embodiments of an RVE system 100 may beimplemented. FIG. 16 illustrates an example computer system that may beused in embodiments of an RVE system 100.

In at least some embodiments, an RVE environment as illustrated in FIG.1B may include an RVE system 100 and one or more client devices 180. TheRVE system 100 has access to stores or other sources of pre-rendered,pre-recorded video, shown as video source(s) 150. The video content mayinclude one or more of, but is not limited to movies, shorts, cartoons,commercials, and television and cable programs. The video available fromvideo source(s) 150 may, for example, include fully 3D rendered,animated video content, as well as partially 3D rendered video contentthat involves filming live action using green- or blue-screen technologyand adding background and/or other content or effects using one or more3D computer graphics techniques.

Note that, in addition to sequences of video frames, a video maytypically include other data such as audio tracks and video metadata.For example, in some embodiments, each frame may have or may correspondto a frame tag that includes information about the frame. The videometadata may include, but is not limited to, time stamps for frames andscene information.

In at least some embodiments, the RVE system 100 may also have access tostores or other sources of data and information including but notlimited to 3D graphics data, shown as data source(s) 160. The 3Dgraphics data may include data that was used in generating and renderingscenes for at least some of the pre-recorded video available from videosources 150, and may also include additional 3D graphics data. Datasource(s) 160 may also store or otherwise provide other data andinformation including but not limited to data and information aboutparticular users 190. Non-limiting examples of user data that may beavailable from data source(s) 160 include RVE system 100 registrationinformation, client device 180 information, name, account number,contact information, billing information, and security information. Insome embodiments, data source(s) 160 may also store or otherwise provideinformation for users including preferences, viewing history, shoppinghistory, sex, age, location, and other demographic and historicalinformation. Note that, while video source(s) 150 and data source(s) 160are shown as separate sources in FIG. 1B, video and data may be obtainedfrom the same source or sources or from different sources.

In at least some embodiments, the RVE system 100 may include a videoplayback 106 module or component and an RVE system interface 102. In atleast some embodiments, RVE system interface 102 may be or may includeone or more application programming interfaces (APIs) for receivinginput from and sending output to RVE client(s) 182 on client device(s)180. In at least some embodiments, in response to user 190 selection ofa video for playback, the video playback 106 module may obtainpre-rendered, pre-recorded video from a video source 150, process thevideo as necessary, and stream the pre-recorded video to the respectiveclient device 180 via RVE system interface 102. Alternatively, the RVEsystem 100 may begin playback of a pre-recorded video, for exampleaccording to a program schedule, and one or more users 190 may choose toview the playback of the video via respective client devices 180.

In at least some embodiments, the RVE system 100 may also include a 3Dgraphics processing and rendering 108 module or component. Note that insome embodiments, 3D graphics processing and 3D rendering may beimplemented as separate components or modules. During an RVE event inwhich the user 190 pauses a video being played back and steps into ascene, 3D graphics processing and rendering 108 module may obtain 3Ddata from one or more data sources 160, generate a 3D modeled world forthe scene according to the 3D data, render 2D representations of the 3Dmodeled world from user-controlled camera viewpoints, and stream thereal-time rendered video to the respective client device 180 via RVEsystem interface 102.

In at least some embodiments, the RVE system 100 may also include an RVEcontrol module 104 that receives input and interactions from an RVEclient 182 on a respective client device 180 via RVE system interface102, processes the input and interactions, and directs operations ofvideo playback module 106 and 3D graphics processing and rendering 108module accordingly. In at least some embodiments, the input andinteractions may be received according to an API provided by RVE systeminterface 102. RVE control module 104 may also track operations of videoplayback module 106 and 3D graphics processing and rendering 108 module.For example, RVE control module 104 may track playback of a given videothrough video playback 106 module so that the RVE control module 104 candetermine which scene is currently being played back to a given clientdevice 180.

In at least some embodiments, RVE system 100 may be implemented by or onone or more computing devices, for example one or more server devices orhost devices, that implement the modules or components 102, 104, 106,and 108, and may also include one or more other devices including butnot limited to storage devices that store pre-recorded video, 3Dgraphics data, and/or other data and information that may be used by RVEsystem 100. FIG. 16 illustrates an example computer system that may beused in some embodiments of an RVE system 100. In some embodiments, thecomputing devices and storage devices may be implemented asnetwork-based computation and storage resources, for example asillustrated in FIG. 13.

However, in some embodiments, functionality and components of RVE system100 may be implemented at least in part on one or more of the clientdevices 180. For example, in some embodiments, at least some clientdevices 180 may include a rendering component or module that may performat least some rendering of video data streamed to the client devices 180from RVE system 100. Further, in some embodiments, instead of an RVEsystem implemented according to a client-server model or variationthereof in which one or more devices such as servers host most or all ofthe functionality of the RVE system, an RVE system may be implementedaccording to a distributed or peer-to-peer architecture. For example, ina peer-to-peer architecture, at least some of the functionality andcomponents of an RVE system 100 as shown in FIG. 1B may be distributedamong one, two, or more devices 180 that collectively participate in apeer-to-peer relationship to implement and perform real-time videoexploration methods as described herein.

While FIG. 1B shows a single client device 180 and client 190interacting with RVE system 100, in at least some embodiments RVE system100 may support many client devices 180. For example, in at least someembodiments, the RVE system 100 may be a network-based video playbackand exploration system that leverages network-based computation andstorage resources to support tens, hundreds, thousands, or even moreclient devices 180, with many videos being played back and/or exploredby different users 190 via different client devices 180 at the sametime. In at least some embodiments, the RVE system 100 may beimplemented according to a service provider's provider networkenvironment, for example as illustrated in FIGS. 13 and 15, that mayimplement one or more services that can be leveraged to dynamically andflexibly provide network-based computation and/or storage resources tosupport fluctuations in demand from the user base. In at least someembodiments, to support increased demand, additional computation and/orstorage resources to implement additional instances of one or more ofthe modules of the RVE system 100 (e.g., 3D graphics processing andrendering module 108, video playback 106 module, RVE control 104 module,etc.) or other components not shown (e.g., load balancers, routers,etc.) may be allocated, configured, “spun up”, and brought on line. Whendemand decreases, resources that are no longer needed can be “spun down”and deallocated. Thus, an entity that implements an RVE system 100 on aservice provider's provider network environment, for example asillustrated in FIGS. 13 and 15, may only have to pay for use ofresources that are needed, and only when they are needed.

FIG. 1C illustrates an example RVE client system, according to at leastsome embodiments. An RVE client system may include a client device 180that implements an RVE client 182. The RVE client 182 may implement anRVE client interface 184 via which the RVE client 182 on device maycommunicate with an RVE system interface 102 of RVE system 100, forexample according to an API or APIs provided by RVE system interface102. The RVE client 182 may receive video stream input from RVE system100 via RVE client interface 184 and send the video to a display 186component of client device 180 to be displayed for viewing. The RVEclient 182 may receive input/interactions from an RVE controls 188component and communicate at least some of the input/interactions to RVEsystem 100 via RVE client interface 184.

A client device 180 may be any of a variety of devices (or combinationsof devices) that can receive, process, and display video input accordingto an RVE client 182 implementation on the device. A client device 180may include, but is not limited to, input and output components andsoftware (RVE client 182 and interface 184) via which users 190 caninterface with the RVE system 100 to play back video and to explorescenes in the video in real-time as described herein. A client device180 may implement an operating system (OS) platform that is compatiblewith the device 180. The RVE client 182 and interface 184 on aparticular client device 180 may be tailored to support theconfiguration and capabilities of the particular device 180 and the OSplatform of the device 180. Examples of client devices 180 may include,but are not limited to, set-top boxes coupled to video monitors ortelevisions, cable boxes, desktop computer systems, laptop/notebookcomputer systems, pad/tablet devices, smartphone devices, game consoles,and handheld or wearable video viewing devices. Wearable devices mayinclude, but are not limited to, glasses or goggles and “watches” or thelike that are wearable on the wrist, arm, or elsewhere. An examplecomputing device that may be used as a client device 180 is illustratedin FIG. 16. Examples of RVE client systems and devices 180 aregraphically illustrated in FIGS. 7A through 12.

In addition to the ability to receive and display video input, a clientdevice 180 may include one or more integrated or external controldevices and/or interfaces that may implement RVE controls 188. Examplesof control devices that may be used include, but are not limited to,conventional cursor control devices such as keyboards and mice,touch-enabled display screens or pads, game controllers, remote controlunits or “remotes” such as those that commonly come with consumerdevices, and “universal” remote control devices that can be programmedto operate with different consumer devices. In addition, someimplementations may include voice-activated interface and controltechnology. Example RVE control interfaces may include, but are notlimited to, control bars or control windows that may be shown/hidden atthe bottom of (or elsewhere on) a video display, and that may beinteracted with via touch devices, cursor control devices, or remotecontrol devices. Note, however, that in some implementations touchgesture input to a video displayed on a touch-enabled device may be usedas RVE controls. Example RVE controls 188 that may be implemented on orby a control device and/or control interface may include one or more of,but are not limited to: pause/resume control(s) for pausing and resumingvideo playback; step in/out control(s) for stepping into or out of aparticular scene; “explore” controls for moving the user's viewpoint or“camera” around (e.g., backwards, forwards, up, down, left right) in ascene, changing the angle of the user's viewpoint, and so on; one ormore controls for selecting objects in the scene, and for manipulatingobjects in the scene in one or more ways; and in general any othercontrols that may be used in controlling video playback and exploring,interacting with, modifying, and manipulating video content includingobjects in a scene.

Note that, in FIGS. 1A and 1B and elsewhere in this document, the terms“user”, “viewer”, or “consumer” are generally used to refer to an actualhuman that participates in an RVE system environment via a client deviceto play back and explore videos as described herein, while the term“client” (as in “client device” and “RVE client”) is generally used torefer to a hardware and/or software interface via which the user orviewer interacts with the RVE system to play back and explore videos asdescribed herein.

As an example of operations of an RVE system 100 as illustrated in FIGS.1A and 1B, RVE control module 104 may direct video playback module 106to begin playback of a selected video or portion thereof from a videosource 150 to a respective client device 180 in response to inputreceived from the client device 180. During playback of the video to theclient device 180, additional input and interactions received by RVEcontrol module 104 from the RVE client 182 on the client device 180 mayindicate an RVE event in which the user 190 pauses the video beingplayed back to the client device 180 and steps into a scene. Inresponse, the RVE control module 104 may direct video playback 106module to pause playback of the pre-recorded video from video source(s)150, and direct 3D graphics processing and rendering 108 module to begingenerating a 3D modeled world for the scene according to 3D data for thescene obtained from data source(s) 160, rendering a 2D representationsof the 3D modeled world, and streaming the real-time rendered video tothe respective client device 180. In response to additional user inputand interactions received from RVE client 182 indicating that the useris exploring the scene, the RVE control module 104 may direct 3Dgraphics processing and rendering 108 module to render and stream newvideo of the scene from the 3D modeled world according to the 3D datafor the scene and current user input, for example new video renderedfrom a particular position and angle within the 3D modeled world of thescene that is indicated by the user's current input to RVE client 182.In response to resume input received from RVE client 182, the RVEcontrol module 104 may direct 3D graphics processing and rendering 108module to stop generating and streaming new exploratory video of thescene, and direct video playback 106 module to resume playback of thepre-recorded video from video source(s) 150.

FIG. 2 is a flowchart of a method for exploring 3D modeled worlds inreal-time during playback of pre-recorded video according to at leastsome embodiments, and with reference to FIGS. 1A and 1B. As indicated at200, an RVE system 100 may begin playback of a pre-recorded video to atleast one client device 180. For example, an RVE control module 104 ofthe RVE system 100 may direct a video playback module 106 to beginplayback of a selected video from a video source 150 to a client device180 in response to selection input received from the client device 180.Alternatively, the RVE system 100 may begin playback of a pre-recordedvideo from a video source 150, and then receive input from one or moreclient devices 180 joining the playback to view (and possibly explore)the video content.

During playback of the pre-recorded video to the client device 180,additional input and interactions may be received by the RVE system 100from an RVE client 182 on a client device 180. For example input may bereceived that indicates an RVE event in which the user 190 pauses thepre-recorded video being played back to the client device 180 so thatthe user 190 can explore the current scene. As indicated at 202, the RVEsystem 100 may continue to play back the pre-recorded video to theclient device 180 until the video is over as indicated at 204, or untilRVE input is received from the client device 180 that directs the RVEsystem 100 to pause the video. At 202, if RVE input requesting a pauseof the video is received from a client device 180, the RVE system 100pauses the replay of the video to the client device 180 at a currentscene, as indicated at 206.

As indicated at 208, while the playback of the pre-recorded video ispaused at a scene, the RVE system 100 may obtain and process 3D data torender new video of the scene in response to exploration input from theclient device 180, and may stream the newly rendered video of the sceneto the client device as indicated at 210. In at least some embodiments,the RVE system 100 may begin generating a 3D modeled world for the scenefrom the 3D data, rendering a 2D representations of the 3D modeledworld, and streaming the real-time rendered video to the respectiveclient device 180 in response to the pause event as indicated at 202 and206. Alternatively, the RVE system 100 may begin generating a 3D modeledworld for the scene from the 3D data, rendering a 2D representations ofthe 3D modeled world, and streaming the real-time rendered video to therespective client device 180 upon receiving additional exploratory inputreceived from the client device 180, for example input changing theviewing angle of the viewer in the scene, or input moving the viewer'sviewpoint through the scene. In response to additional user input andinteractions received from the client device 180 indicating that theuser is further exploring the scene, the RVE system 100 may render andstream new video of the scene from the 3D modeled world according to thecurrent user input and 3D data, for example new video rendered from aparticular position and angle within the 3D modeled world of the scenethat is indicated by the user's current input to the client device 180.Alternatively, in some embodiments, the video may not be paused at 206,and the method may perform elements 208 and 210 while the videocontinues playback.

In at least some embodiments, in addition to allowing users to pause,step into, move through, and explore a scene in a pre-recorded videobeing played back, the RVE system 100 may allow a user to modify thescene, for example by adding, removing, or modifying graphics effectssuch as lens effects (e.g., fisheye, zoom, etc.), lighting effects(e.g., illumination, reflection, shadows, etc.), color effects (colorpalette, color saturation, etc.), or various simulated effects (e.g.,rain, fire, smoke, dust, fog, etc.) to the scenes.

As indicated at 212, the RVE system 100 may continue to render andstream new video of the scene from the 3D modeled world in response toexploratory input until input is received from the client deviceindicating that the user wants to resume playback of the pre-recordedvideo. As indicated at 214, upon receiving resume playback input, theRVE system may resume playing back the pre-recorded video to the clientdevice 180. The playback may, but does not necessarily, resume at thepoint where the playback was paused at 206.

FIGS. 7A through 7C graphically illustrate exploring a rendered 3D modelof a scene while a pre-recorded video is paused, according to at leastsome embodiments. These Figures show, as an example of a client device180, a touch-enabled consumer device 700, such as a tablet or smartphonedevice. The device 700 includes a touch-enabled display screen 702 towhich a rendered scene 704 may be displayed. Initially, scene 704 may bedisplayed from a pre-recorded video being played back to device 700 froman RVE system 100. In FIG. 7A, the user may interact with an RVE controlmethod implemented by the RVE client on device 700 to pause the video atscene 704. For example, a control window may be displayed, and the usermay select a “pause” interface element from the window. Alternatively, atouch gesture may be used to pause the video at the scene 704. Forexample, a double tap on the video display may pause the video at thescene. Other methods may be used to pause a video in variousembodiments. Alternatively, in some embodiments, the video may not bepaused, and the methods may be performed while the video continuesplayback.

In FIG. 7B, the user changes the current viewing angle to view the scene704 from a slightly different angle. In this example, the user uses aright-to-left swipe gesture on the touch-enabled screen to change theviewing angle. However, note that other touch gestures or interfacemethods may be used to change the viewing angle in various embodimentsor client implementations. In response to the user's input, the RVEsystem may render new video of the scene from the changed viewing angleand stream the newly rendered video of the scene to the device 700 fordisplay. In FIG. 7C, the user changes the position of the currentviewpoint to move within the scene 704. In this example, the user uses adownward swipe gesture on the touch-enabled screen to move forward inthe scene 704. However, note that other touch gestures or interfacemethods may be used to move within a scene in various embodiments orclient implementations. In response to the user's input, the RVE systemmay render new video of the scene from the new positions and stream thenewly rendered video of the scene to the device 700 for display.

In at least some embodiments, the RVE system 100 may leveragenetwork-based computation resources and services (e.g., a streamingservice) to receive the user input/interactions from within scene 704 ondevice 700, responsively generate or update a 3D model from the 3D datain response to the user input/interactions, render the new video contentof the scene from the 3D model, and deliver the newly rendered videocontent (and possibly also audio) to the device 700 in real-time ornear-real-time as a video stream. The computational power availablethrough the network-based computation resources, as well as the videoand audio streaming capabilities provided through a streaming protocol,may allow the RVE system 100 to provide low-latency responses to theuser's interactions with the 3D world of the scene 704 as viewed on thedevice 700, thus providing a responsive and interactive exploratoryexperience to the user.

Real-time Object Manipulation in Video Content

At least some embodiments of a real-time video exploration (RVE) system10 such as RVE system 100 shown in FIG. 1B may implement methods thatallow users to discover, select, explore, and manipulate objects withinthe 3D modeled worlds used to generate video content (e.g., scenes inmovies or other video). Leveraging network-based computation resourcesand services and utilizing the rich 3D content and data that was used togenerate and render the original, previously rendered and recordedvideo, an RVE system 100 may allow a viewer of a video, for example amovie, to pause and “step into” a 3D rendered scene from the video via aclient device, for example a device 180 as illustrated in FIG. 1C, todiscover, select, explore, and manipulate objects within the scene. Forexample, a viewer can pause a movie at a scene and interact with one ormore 3D-rendered object(s) in a scene. The viewer may select a 3D modelof an object in the scene, pull up information on or relevant to theselected object, visually explore the object, and in general manipulatethe object in various ways.

FIG. 3 is a flowchart of a method for interacting with objects andrendering new video content of the manipulated objects while exploring apre-recorded video being played back, according to at least someembodiments, and with reference to FIGS. 1A and 1B. As indicated at 300,the RVE system 100 may pause playback of a pre-recorded video beingplayed back to a client device 180 in response to input received fromthe client device 180 to manipulate an object in a scene. In at leastsome embodiments, the RVE system 100 may receive input from the clientdevice 180 selecting an object in a scene displayed on the device 180.In response, the RVE system 100 may pause the pre-recorded video beingplayed back, obtain 3D data for the selected object, generate a 3Dmodeled world for the scene including a new 3D model of the objectaccording to the obtained data, and render and stream new video of thescene to the client device 180.

As indicated at 302, the RVE system 100 may receive input from theclient device 180 indicating that the user is interacting with theselected object via the device 180. As indicated at 304, in response tothe interactive input, the RVE system 100 may render and stream newvideo of the scene from the 3D modeled world including the 3D model ofthe object as manipulated or changed by the interactive input to theclient device 180.

As indicated at 306, optionally, the RVE system 100 may obtain andprovide information for a selected object to the client device 180 inresponse to a request for information. For example, in some embodiments,a user may double-tap on, right-click on, or otherwise select, an objectto display a window of information about the object. As another example,in some embodiments, a user may double-tap on, or right-click on, aselected object to bring up a menu of object options, and select a“display info” option from the menu to obtain the object information.

As indicated at 308, the RVE system 100 may continue to render andstream new video of the scene in response to interactive input withobject(s) in the scene. In at least some embodiments, the RVE system 100may continue to render and stream new video of the scene until input isreceived from the client device indicating that the user wants to resumeplayback of the pre-recorded video. As indicated at 310, upon receivingresume playback input, the RVE system may resume playing back thepre-recorded video to the client device 180. The playback may, but doesnot necessarily, resume at the point where the playback was paused at300.

FIGS. 8A and 8B graphically illustrate selecting and interacting withobjects in a scene when exploring a rendered 3D model of the scene,according to at least some embodiments. FIG. 8A graphically illustratesselecting an object and obtaining information about the object whileexploring a rendered 3D model of a scene, according to at least someembodiments. FIG. 8B graphically illustrates manipulating a rendering ofa selected object while exploring a rendered 3D model of a scene,according to at least some embodiments. These Figures show, as anexample of a client device 180, a touch-enabled consumer device 800,such as a tablet or smartphone device. The device 800 includes atouch-enabled display screen 802 to which a rendered scene 804 may bedisplayed. Initially, scene 804 may be displayed from a pre-recordedvideo being played back to device 800 from an RVE system 100. The usermay interact with an RVE control method implemented by the RVE client ondevice 800 to pause the video at a scene 804. For example, a controlwindow may be displayed, and the user may select a “pause” interfaceelement from the window. Alternatively, a touch gesture may be used topause the video at the scene 804. For example, a double tap on the videodisplay may pause the video at the scene. Other methods may be used topause a video in various embodiments. Note that, in some embodiments, atouch gesture with the screen 802 selecting an object in the scene 804while the pre-recorded video is still being streamed may also cause theRVE system 100 to pause the video.

In FIG. 8A, the user has selected an object 810 in the scene 804, andthe scene 804 is paused. In this example, the selected object 810 is aconsumer electronic device such as a smartphone, PDA, or tablet device.However, note that the object 810 may be virtually anything that can berendered from a 3D model. Non-limiting examples of objects that can bemodeled within scenes, selected, and manipulated by embodiments includefictional or real devices or objects such as vehicles (cars, trucks,motorcycles, bicycles etc.), computing devices (smartphones tabletdevices, laptop or notebook computers, etc.), entertainment devices(televisions and stereo components, game consoles, etc.), toys, sportsequipment, books, magazines, CDs/albums, artwork (painting, sculptures,etc.) appliances, tools, clothes, and furniture; fictional or realplants and animals; fictional or real persons or characters; packaged orprepared foods, groceries, consumables, beverages, and so on; healthcare items (medicines, soap, shampoo, toothbrushes, toothpaste, etc.);and in general any living or non-living, manufactured or natural, realor fictional object, thing, or entity.

Still referring to FIG. 8A, optionally, the user may interact with theobject 810 to obtain more information about the object 810. The RVEsystem 100 may obtain and provide information for the selected object810 to the client device 800 in response to the request for information.For example, in some embodiments, the user may double-tap on, orotherwise select, the object 810 to display a window 806 of informationabout the object. As another example, in some embodiments, a user maydouble-tap on a selected object 806 to bring up a menu of objectoptions, and select a “display info” option from the menu to obtain theobject information

Non-limiting examples of information on or relevant to a selected objectthat may be provided for a selected object 810 may include descriptiveinformation associated and possibly stored with the 3D model data orwith the video being played back. In addition, the information mayinclude, or may include links to, informational or descriptive webpages, advertisements, manufacturer or dealer web sites, reviews, BLOGs,fan sites, and so on. In general, the information that may be madeavailable for a given object may include any relevant information thatis stored with the 3D model data for the object or with the video,and/or relevant information from various other sources such as web pagesor web sites. Note that an “object options” display as shown in FIG. 8Amay include various options for manipulating a selected object 810, forexample options to change color, texture, or other rendered features ofthe selected object 810. At least some of these options may be specificto the type of object.

Referring to FIG. 8B, the user may interact with the selected object 810as displayed on screen 802 using one or more user interface techniques(e.g., touch gesture techniques) to manipulate the object in variousways. For example, as shown in FIG. 8B, the user may use a touch gestureor other interface method to pick up and/or move the object 810, rotatethe object on one or more axes, and so on. Non-limiting examples ofmanipulations of a selected object 810 may include picking up an object,moving an object in the scene, rotating an object as if the object washeld in the viewer's hands, manipulating movable parts of the object, orin general any physical manipulation of the object that can be simulatedvia 3D rendering techniques. Other examples of manipulations of anobject may include changing the rendering of an object such as changingthe lighting, texture, and/or color of the object, changing the opacityof the object so that the object is somewhat transparent, and so on.Other examples of object manipulations may include opening and closingdoors in a house or on a vehicle, opening and closing drawers onfurniture, opening and closing the, trunk, or other compartments on avehicle, or in general any physical manipulation of components of anobject that can be simulated via 3D rendering techniques. As just onenon-limiting example, a user may step into a scene of a paused video toview a vehicle in the scene from all angles, open the doors and goinside the vehicle, open the console or glove compartment, and so on.

In some embodiments, when an object 810 is selected for manipulation, orwhen particular manipulations are performed on the selected object bythe user 810 via the RVE control interface, the RVE system 100 mayaccess additional and/or different 3D graphics applications and/or applyadditional or different 3D graphics techniques than were originally usedto generate and render the object 810 in the scene 804 of the videobeing played back, and may render the object 810 for exploration andmanipulations according to the different applications and/or techniques.For example, the RVE system 100 may use additional or differenttechniques to add or improve texture and/or illumination for an object810 being rendered for exploration and manipulation by the user.

In some embodiments, when an object 810 is selected for manipulation, orwhen particular manipulations are performed on the selected object bythe user, the RVE system 100 may access a different 3D model of theobject 810 than the 3D model that was originally used to generate andrender the object in the scene 804 of the video being played back, andmay render a 3D representation of the object 810 from the different 3Dmodel for exploration and manipulation by the user. The different 3Dmodel may be a more detailed and richer model of the object 810 than theone originally used to render the scene 804, and thus may provide finerdetail and a finer level of manipulation of the object 810 than wouldthe less detailed model. As just one non-limiting example, a user canstep into a scene of a paused video to view, select, and explore avehicle in the scene. In response to selection of the vehicle forexploration and/or manipulation, the RVE system 100 may go to thevehicle's manufacturer site or to some other external source to accessdetailed 3D model data for the vehicle, which may then be rendered toprovide the more detailed 3D model of the vehicle to the user ratherthan the simpler, less detailed, and possibly less current or up-to-datemodel that was used in originally rendering the video.

Still referring to FIG. 8B, at least some embodiments of an RVE system100 may implement methods that allow users to view and explore in moredetail the features, components, and/or accessories of selected objects(e.g., object 810) that are being manipulated and explored. For example,a user may be allowed to zoom in on a selected object 810 to viewfeatures, components, and/or accessories of the selected object 810 ingreater detail. As simple, non-limiting examples, a viewer may zoom inon a bookshelf to view titles of books, or zoom in on a table to viewcovers of magazines or newspapers on the table. As another non-limitingexample, a viewer may select and zoom in on an object such as a notepad,screen, or letter to view the contents in greater detail, and perhapseven to read text rendered on the object. As another non-limitingexample as shown in FIGS. 8A and 8B, a computing device that is renderedin the background of a scene and thus not shown in great detail may beselected, manipulated, and zoomed in on to view fine details on thedevice's screen or of the device's accessories and interface componentssuch as buttons, switches, ports, and keyboards, or even model or partnumbers. As another non-limiting example, an automobile that is renderedin the background of a scene and thus not shown in great detail may beselected, manipulated, and zoomed in on to view fine details of theoutside of the automobile. In addition, the viewer may open the door andenter the vehicle to view interior components and accessories such asconsoles, navigation/GPS systems, audio equipment, seats, upholstery,and so on, or open the hood of the vehicle to view the enginecompartment.

In addition to allowing users to select and manipulate objects in ascene as described above, at least some embodiments of an RVE system 100may implement methods that allow users to interact with interfaces ofselected objects or interfaces of components of selected objects. As anexample of a device and interactions with a device that may be simulatedby RVE system 100, a viewer may be able to select a rendered objectrepresenting a computing or communications device such as a cell phone,smart phone, tablet or pad device, or laptop computer, and interact withthe rendered interface of the device to simulate actual operations ofthe device. As another example of a device and interactions with adevice that may be simulated by RVE system 100, a user may enter anautomobile rendered on the client device 180 and simulate operations ofa navigation/GPS system in the automobile's console via the renderedrepresentation of the navigation/GPS system's interface. The renderedobject may respond appropriately to the user's interactions, for exampleby appropriately updating a touchscreen in response to a swipe or tapevent. Reactions of a rendered object in response to the user'sinteractions via the rendered interface may, for example, be simulatedby the RVE system 100 according to the object type and object data, ormay be programmed, stored with, and accessed from the object's 3D modeldata or other object information.

FIG. 9 graphically illustrates interacting with components andinterfaces of a selected object while exploring a rendered 3D model of ascene, according to at least some embodiments. FIG. 9 shows, as anexample of a client device 180, a touch-enabled consumer device 900,such as a tablet or smartphone device. The device 900 includes atouch-enabled display screen 902 to which a rendered scene 904 includingan object 910 with an interface may be displayed. In FIG. 9, the userhas selected the object 910 in the scene 904, and the scene 904 ispaused. The user has also used one or more user interface techniques(e.g., touch gestures) to zoom in on the object 910 to show the object910 in greater detail. In this example, the selected object 910 is aconsumer electronic device 910 such as a smartphone, PDA, or tabletdevice. However, the object 910 may be any type of object, or componentof an object, with an interface. As examples of interactions with aninterface of an object 910, on a rendered representation of atouchscreen 912 interface of a device 910 such as a tablet orsmartphone, the user may be allowed to tap on the touchscreen 912,select icons, open applications, perform various touch operations suchas swiping and tapping, view and enter text, and otherwise simulateactual operations of the device through the interface displayed on thescreen 902 of client device 900. The user may also interact with atleast some other “physical” controls of the device 910 such as buttons,switches, or other interface components that are rendered for the device910 on the screen 902. The rendered object 910 may respond appropriatelyto the user's interactions, for example by appropriately updating thetouchscreen 912 in response to a swipe or tap event. Reactions of therendered object 910 in response to the user's interactions via therendered interface may, for example, be simulated by the RVE system 100according to the object type/model and object data accessed by the RVEsystem 100, or may be programmed, stored with, and accessed from theobject's 3D model data or other object information local to RVE system100 or on a remote site such as a manufacturer's site.

Referring to FIGS. 8A, 8B, and 9, in at least some embodiments, an RVEsystem 100 may leverage network-based computation resources and services(e.g., a streaming service) to receive the user's manipulations ofobjects in scenes on a client device, responsively generate or update 3Dmodels of the scenes with modified renderings of the manipulated objectsin response to the user input, render new video of the scenes, anddeliver the newly rendered video to the client device in real-time ornear-real-time as a video stream. The computational power availablethrough the network-based computation resources, as well as the videoand audio streaming capabilities provided through a streaming protocol,may allow the RVE system 100 to provide low-latency responses to theuser's interactions with the objects in a scene, thus providingresponsive and interactive manipulations of the objects to the user.

Other RVE Client System Implementations

FIGS. 7A through 9 show, as an example of a client device 180, aconsumer device with a touchscreen interface such as a tablet orsmartphone device. However, other client devices 180 and interfaces maybe used in embodiments. FIGS. 10 and 11 show some non-limiting examplesof other RVE client devices that may be used, and also furtherillustrate aspects of scene exploration and object manipulation in anRVE environment.

FIG. 10 illustrates an example RVE system and environment in which aclient device 1000 uses an external control device 1022 to explore videocontent, according to at least some embodiments. FIG. 10 alsographically illustrates selecting an object and obtaining informationabout the object while exploring a rendered 3D model of a scene,according to at least some embodiments. Client device 1000 may include adisplay 1002 and an external control device 1022 coupled (via wired orwireless connections) to a central unit 1020. Central unit 1020 may, forexample, be a game controller, a set-top box, a desktop or othercomputer, a cable box or modem, or in general any device or unit thatcan communicate with RVE system 100, display 1002, and control device1022. An example computing device that may be used as a central unit1020 is illustrated in FIG. 16. Display 1002 may, for example, be avideo monitor or television set of any of various types, or may be adisplay screen of a computing device such as a laptop or desktopcomputer or tablet device, or of some other device. Display 1002 may beexternal to central unit 1020 as shown, or alternatively may beintegrated with central unit 1020. External control device 1022 may, forexample, be a game controller or a conventional remote control unit or“remote” that may be provided with, or programmed to operate with,different consumer devices.

Video streamed from RVE system 100 to device 1000 may be received atcentral unit 1020, processed, and displayed to display 1002. Initially,the streamed video may be a pre-recorded video being played back todevice 1000 from the RVE system 100. Via the remote control device 1022,the user may interact with an RVE control method implemented by the RVEclient on device 1000 to pause the video at a scene 1004. For example, acontrol window may be displayed, and the user may select a “pause”interface element from the window via device 1022. Alternatively, device1022 may have a “pause” button or other interface element that may beselected to pause the video at the scene 1004. Other methods may be usedto pause a video in various embodiments. The user may then use remotecontrol device 1022 to explore the scene 1004 (e.g., change viewingangles, changes positions, etc.) and to select and manipulate objects,such as object 1010, as described herein. In response to userexploration, selection, and manipulation input to remote control device1022, the RVE system 100 may, if necessary obtain additional 3D data foraccessorizing or modifying the selected object 1010, for example fromone or more external sources, and may generate, render, and stream anupdated view of scene reflecting the user input.

In FIG. 10, the user has selected object 1010 in the scene 1004, and thescene 1004 is paused. In this example, the selected object 1010 is aconsumer electronic device such as a smartphone, PDA, or tablet device.However, note that the object 1010 may be virtually anything that can berendered from a 3D model. In at least some embodiments, the user mayinteract with the displayed object 1010 via device 1022 to obtain moreinformation about the object 1010. The RVE system 100 may obtain andprovide information for the selected object 1010 to the client device1000 in response to the request for information. For example, in someembodiments, the user may use device 1022 to select the displayed object1010; in response to the selection, a window 1030 of information aboutthe object 1010 may be displayed to display 1002. Options formanipulating or modifying the object 1010 may also be displayed towindow 1030.

FIG. 11 illustrates an example RVE system and environment in which anRVE client device 1100 uses a “second screen” 1122 to explore the videocontent, according to at least some embodiments. FIG. 11 alsographically illustrates selecting an object and obtaining informationabout the object while exploring a rendered 3D model of a scene,according to at least some embodiments. Client device 1100 may include afirst display 1102 and a second display 1122 or “second screen” coupled(via wired or wireless connections) to a central unit 1120. Videostreamed from RVE system 100 to device 1100 may be received at centralunit 1120, processed, and displayed to display 1102. Initially, thestreamed video may be a pre-recorded video being played back to device1100 from the RVE system 100. Via RVE controls 188 (see, e.g., FIG. 1C)displayed on second screen 1122, the user may pause the video at a scene1104. The user may then use RVE controls 188 on second screen 1122 toexplore the scene 1104 (e.g., change viewing angles, changes positions,etc.) and to select and manipulate objects, such as object 1110, asdescribed herein. In some implementations, second screen 1122 may betouch-enabled, and thus the user may interact with the interfacedisplayed on the device 1122 using touch gestures. Instead or inaddition, device 1100 may also include external cursor control devicessuch as a keyboard and mouse or external control devices such as a gamecontroller or remote controller via which the user may interact with thedisplayed interface on second screen 1122.

In FIG. 11, the user has selected object 1110 in the scene 1104 usingRVE controls 188, and the scene 1104 is paused. In at least someembodiments, the user may interact with the displayed object 1110 viaRVE controls 188 on second screen 1122 to obtain more information aboutthe object 1110. The RVE system 100 may obtain and provide informationfor the selected object 1110 to the client device 1100 in response tothe request for information. For example, in some embodiments, the usermay use RVE controls 188 on device 1122 to select the displayed object1110; in response to the selection, a window 1130 of information aboutthe object 1110 may be displayed to the second screen 1122 device.Options for manipulating or modifying the object 1110 may also bedisplayed to window 1030. In at least some embodiments, otherinformation 1140 (e.g., information about the video, the scene, etc.)may also be displayed to device 1122.

Real-time Object Modification in Video Content

At least some embodiments of a real-time video exploration (RVE) system10 such as RVE system 100 shown in FIG. 1B may implement methods thatallow users to interact with selected objects to customize oraccessorize the objects. Leveraging network-based computation resourcesand services and utilizing 3D data for rendered objects in a video, anRVE system 100 may allow a viewer of the video, for example a movie, topause and “step into” a 3D rendered scene from the video via a clientdevice, for example a device 180 as illustrated in FIG. 1C, and to,discover, select, explore, and manipulate objects within the scene. Inaddition, for 3D-rendered objects in a scene that can be accessorized orcustomized with options, the viewer can manipulate or interact with aselected object to add or remove accessories, customize the object(change color, texture, etc.), or otherwise modify the object accordingto the user's preferences or desires. As a non-limiting example, a usermay interact with a rendering of an automobile of a scene to accessorizeor customize the car. For example, the user can change the exteriorcolor, change the interior, change the car from a hardtop to aconvertible, and add, remove, or replace accessories such asnavigation/GPS systems, audio systems, special wheels and tires, and soon. In at least some embodiments, and for at least some objects, the RVEsystem 100 may also facilitate pricing, purchasing, or ordering of anobject (e.g., a car) as accessorized or customized by the user via aninterface on the client device.

Since the modifications to an object are done in a 3D-renderedscene/environment, the viewer can customize and/or accessorize an objectsuch as an automobile and then view the customized object as rendered inthe 3D world of the scene, with lighting, background, and so on fullyrendered for the customized object. In at least some embodiments, theuser-modified object may be left in the scene when the video is resumed,and the object as it appears in the original video in this and otherscenes may be replaced with the rendering of the user's modified versionof the object. Using an automobile as an example, the viewer maycustomize a car, for example by changing it from red to blue, or from ahardtop to a convertible, and then view the customized car in the 3Dmodeled world of the scene, or even have the customized car used in therest of the video once resumed.

In at least some embodiments of an RVE system 100, the ability tocustomize and/or accessorize objects may, for at least some objects, belinked to external sources, for example manufacturer, dealer, and/ordistributor information and website(s). The RVE system 100 may providean interface, or may invoke an external interface provided by themanufacturer/dealer/distributor, via which the user can customize and/oraccessorize a selected object if and as desired (e.g., an automobile, acomputing device, an entertainment system, etc.), be given a price orprice(s) for the object as customized/accessorized, and even order orpurchase the object as specified if desired.

FIG. 4 is a flowchart of a method for modifying, and optionallyordering, objects while exploring a video being played back, accordingto at least some embodiments, and with reference to FIGS. 1A and 1B. Asindicated at 400, the RVE system 100 may pause playback of apre-recorded video being played back to a client device 180 in responseto input received from the client device 180 to manipulate an object ina scene. In at least some embodiments, the RVE system 100 may receiveinput from the client device 180 selecting an object in a scenedisplayed on the device 180. In response, the RVE system 100 may pausethe pre-recorded video being played back, obtain 3D data for theselected object, generate a 3D modeled world for the scene including anew 3D model of the object according to the obtained data, and renderand stream new video of the scene to the client device 180.

As indicated at 402, the RVE system 100 may receive input from theclient device 180 indicating that the user is interacting with theselected object via the device to modify (e.g., accessorize orcustomize) the selected object. In response, the RVE system 100 mayobtain additional 3D data for accessorizing or modifying the selectedobject, and generate a new 3D modeled world for the scene including anew 3D model of the object according to the modifications specified bythe user input. As indicated at 404, the RVE system 100 may render andstream new video of the scene from the 3D modeled world including the 3Dmodel of the object as modified by the input to the client device 180.

As shown at 406, optionally, the RVE system 100 may receive additionalinput from the client device 180 requesting additional information aboutthe object as modified (e.g., pricing, availability, vendors, dealers,etc.), and/or additional information indicating that the user wants topurchase or order a physical version of the object as modified (or asoriginally rendered, if desired). In at least some embodiments, inresponse to requests for additional information, the RVE system 100 mayprovide additional object information (e.g., websites, links, emails,documents, advertisements, pricing, reviews, etc.) to the user viaclient device 180. In at least some embodiments, in response to arequest to order or purchase an item, the RVE system 100 may provide aname, location, URL, link, email address, phone number, and/or otherinformation indicating one or more online or brick-and-mortar sourcesfor ordering or purchasing the object. In some embodiments, the RVEsystem 100 may provide a purchasing interface via which the user canorder the object as modified. In at least some embodiments, a user mayorder, purchase, or obtain a virtual representation of the objectinstead of or in addition to a physical version of the object, ifdesired.

As indicated at 408, the RVE system 100 may continue to render andstream new video of the scene in response to interactions with object(s)in the scene. In at least some embodiments, the RVE system 100 maycontinue to render and stream new video of the scene until input isreceived from the client device indicating that the user wants to resumeplayback of the pre-recorded video. As indicated at 410, upon receivingresume playback input, the RVE system may resume playing back thepre-recorded video to the client device 180. The playback may, but doesnot necessarily, resume at the point where the playback was paused at400.

FIG. 12 illustrates an example RVE system and environment in whichobjects in a pre-recorded video can be modified, new video contentincluding the modified objects can be generated and streamed,information on objects can be obtained, and virtual or physical versionsof objects from the video can optionally be ordered, according to atleast some embodiments. In FIG. 12, an RVE client device 1200 uses a“second screen” 1222 to explore the video content, according to at leastsome embodiments. Client device 1200 may include a first display 1202and a second display 1222 or “second screen” coupled (via wired orwireless connections) to a central unit 1220. Central unit 1220 may, forexample, be a game controller, a set-top box, a desktop or othercomputer, a cable box or modem, or in general any device or unit thatcan communicate with RVE system 100 and displays 1202 and 1222. Anexample computing device that may be used as a central unit 1220 isillustrated in FIG. 16.

Video streamed from RVE system 100 to device 1200 may be received atcentral unit 1220, processed, and displayed to display 1202. Initially,the streamed video may be a pre-recorded video being played back todevice 1200 from the RVE system 100. Via RVE controls 188 (see, e.g.,FIG. 1C) displayed on second screen 1222, the user may pause the videoat a scene 1204. The user may then use RVE controls 188 to explore thescene 1204 (e.g., change viewing angles, changes positions, etc.), toselect and manipulate objects, such as object 1210, and to modifyobjects such as object 1210 by accessorizing or otherwise customizingthe objects. In this example, the selected object 1210 is a consumerelectronic device such as a smartphone, PDA, or tablet device. However,note that the object 1210 may be virtually anything that can be renderedfrom a 3D model and that can be manipulated, customized, and/oraccessorized.

In FIG. 12, the user has selected object 1210 in the scene 1204 usingRVE controls 188, and the scene 1204 is paused. In at least someembodiments, the user may interact with the displayed object 1210 viaRVE controls 188 on second screen 1222 to manipulate the object and/orto obtain more information about the object 1210. In at least someembodiments, a detailed rendering of the selected object 1210 may bedisplayed to the second screen 1222. The RVE system 100 may obtaininformation about the selected object 1210, for example from one or moreexternal sources 1250, and may provide the obtained information for theselected object 1210 to the client device 1200 in response to therequest for information. For example, in some embodiments, the user mayuse RVE controls 188 on device 1222 to select the object 1210; inresponse to the selection, information about the object 1210 may bedisplayed to the second screen 1222 device, for example to a window1234, and a detailed rendering of the object 1210 may also be displayed.

In at least some embodiments, one or more accessorization andcustomization options for modifying the object 1210 may be displayed toa window 1232. The user may then use the interface presented on secondscreen 1222 to accessorize or customize the object 1210 according to theavailable options. The object modification input may be received bycentral unit 1220 and forwarded to RVE system 100. In response to theobject modification input, the RVE system 100 may obtain additional 3Ddata for accessorizing or modifying the selected object 1210, forexample from one or more external sources 1250, and generate a new 3Dmodeled world for the scene including a new 3D model of the objectaccording to the modifications specified by the user input. The RVEsystem 100 may then render and stream new video of the scene from the 3Dmodeled world including the 3D model of the object as modified by theinput to the client device 1200. At the client device, the modificationsto the object 1210 may be reflected on the object 1210 displayed on thesecond screen 1222 and/or on the object 1210 displayed in scene 1204.

In at least some embodiments of an RVE system 100, the ability tocustomize and/or accessorize objects may, for at least some objects, belinked to external sources 1250, for example manufacturer, dealer,and/or distributor information and website(s). The RVE system 100 mayprovide an interface, or may invoke an external interface 1234 such as aweb page provided by the manufacturer/dealer/distributor, via which theuser can customize and/or accessorize a selected object if and asdesired (e.g., an automobile, a computing device, an entertainmentsystem, etc.), be given information including but not limited to a priceor price(s) for the object as customized/accessorized, and even order orpurchase a physical and/or virtual version of the object from anexternal source 1250 as specified if desired. In FIG. 12, the interfacefor customizing and/or accessorizing a selected object, obtaininginformation such as pricing about the object, and ordering physicaland/or virtual versions of the object if desired is shown on secondscreen 1222 as object accessorization/customization options window 1232and object information/ordering window 1234.

In at least some embodiments of an RVE system 100, in addition tocustomizing or accessorizing a selected object 1210, a user may beallowed to replace an object 1210 with a different object. In FIG. 12,for example, the selected object 1210 is a consumer electronic devicesuch as a smartphone, PDA, or tablet device, and in some embodiments theuser may be allowed to replace the device with a device of another brandor make. Using an automobile as an example, the viewer may replace onetype of car in the 3D rendered environment with another type of car, andthen view the different car in the 3D modeled world of the scene, oreven have the different car used in the rest of the video once resumed.

Referring to FIG. 12, in at least some embodiments, an RVE system 100may leverage network-based computation resources and services (e.g., astreaming service) to receive the user's modifications of objects inscenes on a client device, responsively generate or update 3D models ofthe scenes with modified renderings of the objects in response to theuser input, render new video of the scenes, and deliver the newlyrendered video to the client device in real-time or near-real-time as avideo stream. The computational power available through thenetwork-based computation resources, as well as the video and audiostreaming capabilities provided through a streaming protocol, may allowthe RVE system 100 to provide low-latency responses to the user'smodifications of the objects in a scene, thus providing responsive andinteractive modifications of the objects to the user.

Real-time Object Modifications from other Sources

FIG. 4 describes embodiments of a method in which the RVE system 100 mayreceive input from a client device 180 indicating that a user viewing avideo is interacting with an object to modify the selected object.However, input may be received from other sources than viewers to modifyobjects in videos in real-time.

For example, the RVE system 100 may store viewer preferences or profilesin a database. The viewer profiles or preferences may be accessedaccording to identities of the viewer(s) when beginning replay of, orduring the replay of, a video (e.g., a movie), and used to dynamicallyand differently render one or more objects in one or more scenes, forexample to target the content at the particular viewers according totheir respective profiles or preferences. The RVT system 100 may streamvideo including the targeted content to the respective client device(s).Thus, different viewers of the same video content (e.g., a movie) may beshown the same scenes with differently rendered objects injected intothe scenes. In some embodiments, a viewer may change their preferencesor profile when viewing a video, and the RVT system 100 may dynamicallyand differently render one or more objects in one or more scenes inresponse to the change(s).

As another example the RVT system 100 may obtain input modifying objectsin video from one or more sources other than the viewers, for examplefrom manufacturer, vendor, dealer, or distributor websites. Themodifications received from external sources may be used to dynamicallyand differently render one or more objects in one or more scenes of oneor more videos. The modifications may, for example, target video contentor objects at particular viewers or groups of viewers for marketing- oradvertising-based placement of particular products based on the viewers'preferences or profiles, or based on other information such asdemographics data.

In at least some embodiments, the graphics data used to modify objectsmay be obtained from a data store maintained by the RVT system 100.However, in at least some embodiments, at least some of the graphicsdata for modifying video content may be obtained from other, externaldata sources, for example from manufacturer, vendor, dealer, ordistributor websites. For example, modifications to a rendered object,or a modified version of the object, may be received from a seller of aphysical version of the rendered object.

Generating New Video Content from Pre-recorded Video

At least some embodiments of a real-time video exploration (RVE) system10 may allow a user to generate their own customized version of a videosuch as a movie. The generated video may be recorded for later playback,or may be streamed or broadcast “live” to other endpoints or viewers.FIG. 6A is a high-level illustration of a real-time video exploration(RVE) system 10 that enables the generation and output of new video frompre-recorded video, according to at least some embodiments. A user mayinteract with video via an RVE client 30 to generate modified orcustomized video from pre-recorded video and graphics data obtained fromone or more sources 20. In at least some embodiments, the RVE system 10may play back video from one or more sources 20 to the RVE client 30,receive user input/interactions within scenes being explored from therespective RVE client 30, responsively generate or update 3D models fromgraphics data obtained from one or more sources 20 in response to theuser input/interactions exploring the scenes, render new video contentof the scenes at least in part from the 3D models, and deliver the newlyrendered video content (and audio, if present) to the respective RVEclient 30 as RVE video.

For example, a user may pause a video being replayed at a scene, changethe viewing angle and/or viewing position for the scene via a userinterface to the RVE system 10 (e.g., RVE controls 188 as shown in FIG.1C), and re-render a portion of or the entire scene using the modifiedviewing angle and/or position, for example using a method as illustratedin FIG. 2. As another example, the user may modify the scene by adding,removing, or modifying various graphics effects such as lens effects,lighting effects, color effects, or various simulated effects to thescene. The user may do this for one or more scenes in a video (e.g., amovie), to generate a new version of the entire video or a portionthereof including the modified and re-rendered views of the scene(s).

As another example, the user may manipulate, modify, customize,accessorize and/or rearrange objects in one or more scenes of a videousing one or more of the methods previously described, for example inFIGS. 3 and 4, and/or remove or add objects to a scene, to generate anew version of the entire video or a portion thereof including themodified object(s) in the scene(s). One or more of these methods, orcombinations of two or more of these methods, may be used to modify orcustomize a given scene or video.

The user may interact with RVE system 10 via RVE client 30 to record,stream, and/or broadcast the new video to one or more destinations 40.The new versions of videos or portions of videos so produced may, forexample, be stored or recorded to local or remote storage, shown to orshared with friends, or may be otherwise recorded, stored, shared,streamed, broadcast, or distributed assuming the acquisition ofappropriate rights and permissions to share, distribute, or broadcastthe new video content. In at least some embodiments, RVE system 10 mayprovide one or more application programming interfaces (APIs) forreceiving input from and sending output to RVE client(s) 30.

FIG. 5 is a flowchart of a method for rendering and storing new videocontent during playback of pre-recorded video, according to at leastsome embodiments, and with reference to FIG. 6A. As indicated at 500, anRVE system 10 may play back at least a portion of a pre-recorded videoto an RVE client 30. As indicated at 502, the RVE system 10 may processand render video of one or more scenes in the video in response to inputfrom the RVE client 30. For example, in at least some embodiments, auser may pause a video being replayed, change the viewing angle and/orviewing position for the scene, and re-render the scene or a portionthereof using the modified viewing angle and/or position, for exampleusing a method as described in FIG. 2 and illustrated in FIGS. 7A and7B. As another example, the user may manipulate, modify, customize,accessorize and/or rearrange objects in one or more scenes, for exampleas described in FIGS. 3 and 4 and illustrated in FIGS. 8A through 12.Note that one or more of these methods, or combinations of two or moreof these methods, may be used to modify a given scene or portions of ascene. As indicated at 504, the RVE system 10 may stream the newlyrendered video of the scene to the RVE client 30. As indicated at 506,at least a portion of the video being played back may be replaced withthe newly rendered video according to input from the RVE client 30. Forexample, one or more scenes in the original video may be replaced withnewly rendered scenes recorded from modified perspectives and/orincluding modified content to generate a new version of the originalvideo. As indicated at 508, at least a portion of the modified video maybe provided to one or more destinations 30 as new video content. Newversions of videos or portions of videos so produced may, for example,be recorded or stored to local or remote storage, shown to or sharedwith friends, or may be otherwise stored, shared, streamed, broadcast,or distributed assuming the acquisition of appropriate rights andpermissions to share or distribute the new video content.

The elements of FIG. 5 are explained below in more detail with furtherreference to FIGS. 6B and 6C.

FIGS. 6B and 6C illustrate example real-time video exploration (RVE)systems 100 in RVE environments in which users can generate, render andstore new video content during playback of a pre-recorded video,according to at least some embodiments. As shown in FIG. 6B, in at leastsome embodiments, an RVE environment may include an RVE system 100 andone or more client devices 180. An example client device 180 that may beused is shown in FIG. 1C. A client device 180 may implement an RVEclient 182 and RVE controls 188. The RVE system 100 has access to storesor other sources of pre-rendered, pre-recorded video, shown as videosource(s) 150. The RVE system 100 also has access to stores or othersources of data and information including but not limited to 3D graphicsdata, shown as data source(s) 160. The 3D graphics data may include, butis not limited to, data that was used in generating and rendering scenesfor at least some of the pre-recorded video available from video sources150, and may also include additional 3D graphics data.

As shown in FIG. 6B, in at least some embodiments, the RVE system 100may include a video playback 106 module or component and an RVE systeminterface 102. In at least some embodiments, RVE system interface 102may be or may include one or more application programming interfaces(APIs) for receiving input from and sending output to RVE client(s) 182on client device(s) 180. In response to user selection of a video forplayback to client device 180, the video playback 106 module may obtainpre-rendered, pre-recorded video from a video source 150, process thevideo as and if necessary, and stream the pre-recorded video to therespective client device 180 via RVE system interface 102. The RVEsystem 100 may also include a 3D graphics processing and rendering 108module or component. During an RVE event in which the user 190 pauses avideo being played back to client device 180, steps into a scene,explores, and possibly modifies video content such as rendered objectsvia RVE client 182, 3D graphics processing and rendering 108 module mayobtain 3D data from one or more data sources 160, generate a 3D modeledworld for the scene according to the obtained 3D data and user input,render 2D representations of the 3D modeled world from user-controlledcamera viewpoints, and stream the real-time rendered video to therespective client device 180 via RVE system interface 102.

As shown in FIG. 6B, in at least some embodiments, the RVE system 100may also include a video output 110 module or component that may recordand/or broadcast new video content generated in the RVE environment toone or more destinations 170. For example, during (or after) an RVEevent in which new video content is generated and rendered frompre-recorded video being played back, video output 110 module mayreceive at least a portion of the real-time rendered video from 3Dgraphics processing and rendering 108 module and record the new video toa video destination 170. In some embodiments, video output 110 modulemay also receive at least a portion of the pre-recorded video beingplayed back through video playback 106 module and merge or combine thereal-time rendered video with the pre-recorded video, for example byreplacing particular scenes or portions thereof in the original,pre-recorded video and recording and/or broadcasting the results as newvideo to one or more destinations 170

As shown in FIG. 6B, in at least some embodiments, the RVE system 100may also include an RVE control module 104 that receives input andinteractions from an RVE client 182 on a respective client device 180via RVE system interface 102, processes the input and interactions, anddirects operations of video playback module 106, 3D graphics processingand rendering 108 module, and new video output 110 module accordingly.In at least some embodiments, the input and interactions may be receivedaccording to an API provided by RVE system interface 102.

In at least some embodiments, user 190 may modify one or more scenes ofa video being played back by video playback 106 module RVE system 100using an RVE controls 188 interface to RVE system 100 as implemented byan RVE client 182 on a client device 180. An example of a client device180 and RVE client 182 are shown in FIG. 1C. FIGS. 7A through 12illustrate several example RVE systems and environments in which clientdevices use different components or devices to implement an RVE controls188 interface. For example, in at least some embodiments, a user 190 maypause a video being replayed via video playback 106 module, change theviewing angle and/or viewing position for the scene via RVE controls188, and re-render the scene or a portion thereof using the modifiedviewing angle and/or position, for example using a method as describedin FIG. 2 and illustrated in FIGS. 7A and 7B. As another example, theuser 190 may manipulate, modify, customize, accessorize and/or rearrangeobjects in one or more scenes, for example as described in FIGS. 3 and 4and illustrated in FIGS. 8A through 12. Note that one or more of thesemethods, or combinations of two or more of these methods, may be used tomodify a given scene or portions of a scene. The user may use thesemethods to modify one, two, or more scenes or portions thereof in avideo (e.g., a movie).

In at least some embodiments, in addition to controls for pausing,exploring, and modifying video content of scenes in a video being playedback from RVE system 100, the RVE controls 188 interface may include oneor more controls 189 via which the user 190 may record and/or broadcastnew video content generated by 3D graphics processing and rendering 108module according to the user's modifications and manipulations of scenesfrom a pre-recorded video (e.g., movie) being played back. In at leastsome embodiments, using controls 189 of the RVE controls 188 interface,the user 190 may be able to selectively specify which parts of a videobeing played back are to be replaced by new video content rendered by 3Dgraphics processing and rendering 108 module. The user 190 may also beable to perform various other recording and/or broadcasting functionsusing controls 189 of the RVE controls 188 interface. As a non-limitingexample, in at least some embodiments, the user 190 may be able tocreate new video content by combining one or more newly rendered scenesor portions of scenes as modified by the user from scenes in one or morevideos.

As an example method of recording new video, in at least someembodiments, a user 190 may change the viewing angle and/or viewingposition for the scene via RVE controls 188, re-render the scene or aportion thereof using the modified viewing angle and/or position, andselect a “record scene” option from RVE controls 188. Instead or inaddition, the user 190 may manipulate, modify, customize, accessorizeand/or rearrange objects in a scene and select a “record scene” optionfrom RVE controls 188. In at least some embodiments, each modified scenethat the user 190 so records may be recorded to one or more destinations170 as new video content by a video output 110 component of RVE system100, for example to a local store of client device 180 or to a remotestore (e.g., video source(s) 150) accessed and provided through RVEsystem 100. In at least some embodiments, the user 190 may direct RVEsystem 100 to combine two or more such scenes into new video contentusing RVE controls 188. In response, video output 110 module of the RVEsystem 100 may combine the scenes into a single, new video segment andstore the new video. In at least some embodiments of an RVE system 100,modified and rendered scenes generated from two or more pre-recordedvideos may be combined to produce new video content.

As another example method of recording new video, in at least someembodiments, a user 190 may modify one or more scenes of a pre-recordedvideo (e.g., a movie) being played back by changing viewpoint positionsand angles and/or by manipulating various object(s), save particularones of the modifications or modified scenes, and then select a “recordnew version of video” option from RVE controls 188. In response, videooutput 110 module may generate and record a new version of the video bycombining new video content rendered by 3D graphics processing andrendering 108 module with video content from the original video. Forexample, one or more scenes or portions thereof in the original videomay be replaced with new versions of the scenes as rendered by 3Dgraphics processing and rendering 108 module.

In at least some embodiments, instead of or in addition to recording newvideo and playing back the recorded new video, the RVE system 100 mayenable the real-time streaming or broadcasting of new video generated bya user via an RVE client 182 as described herein to one, two, or moreother endpoints as destinations 170 for display. An endpoint may, forexample, be another RVE client 182 on another client device 180.However, an endpoint may be any device configured to receive and displaya video stream from RVE system 100. As an example of broadcasting newvideo, in some embodiments a user may use an RVE client 182 on a clientdevice 180 to perform a “video DJ” function in which the user customizesinput video using the RVE system 100 in real-time and broadcasts thecustomized video via the RVE system 100 in real-time to one or moreendpoints, for example one or more local or remote devices configured todisplay video received in streams from RVE system 100.

FIG. 6B shows a video output 110 module implemented by RVE system 100.However, as shown in FIG. 6C, in some embodiments a video output 112module may instead or in addition be implemented by an RVE client 182 ona client device 180. During an RVE event in which new video content isgenerated and rendered by 3D graphics processing and rendering 108module from pre-recorded video being played back by video playback 106module and streamed to the client device 180 through RVE systeminterface 102, video output 112 module on client device 180 may receiveand record at least a portion of the video being streamed from RVEsystem 100 to client device 180 to a store 192. Store 192 may, forexample, be a local store of client device 180, or network-basedstorage. Note that the video being streamed from RVE system 100 toclient device 180 may include real-time rendered video from 3D graphicsprocessing and rendering 108 module as well as pre-recorded video beingplayed back through video playback 106 module.

Example Real-time Video Exploration (RVE) Use Cases

FIG. 2 illustrates a method for exploring a 3D modeled world inreal-time during playback of pre-recorded video by an RVE system 100 inwhich a viewer can pause a video (e.g. a movie), step into a scene, andexplore the scene. In addition, FIG. 3 illustrates a method in which theviewer can manipulate objects and discover more information about objectwithin a paused scene during exploration using the RVE system 100. Thesemethods may be used in some embodiments to provide engaging, interactivevideo experiences in which a user may look for clues, “Easter Eggs”, orother content that may be embedded or concealed in a scene of a movie orother video by the creator, and that may be hidden or not easilydetectable during normal playback of the scene.

As an example, for some types of story lines, for example murdermysteries, a viewer can play back and view the movie as normal using theRVE system 100. However, if the viewer chooses, the viewer can pause thevideo at a scene, step into the scene, and look or search for clues thatmay be hidden or at least not obvious in the pre-rendered scene usingthe RVE system 100. The viewer can explore the scene in more detail andfrom different angles and positions, looking behind and under objects,and manipulating objects to look for clues or further investigate theobjects. For example, there may be a note on a desk, or in a drawer ofthe desk, or even in the pocket of a victim that the viewer can discoverand read. As another example, there may be a text message or voicemessage on a cell phone, or an email message on a computer screen, thatthe user can access by interacting with the respective objects to viewor even listen to. As another example, an object may be hidden under acouch or bed, or in a closet, that the viewer might discover. As anotherexample, clues may be hidden in the trunk of a car, or elsewhere in thecar. As another example, weapons may be discovered, or footprints,fingerprints, or other forensic evidence. The viewer can thus pause,step into, and interact with scenes in a movie being played back topersonally look for clues and investigate a mystery on his or her own.When done, the viewer can resume normal playback of the movie.

As another example, a video content creator may hide one or more “EasterEggs” in a video such as a movie. An “Easter Egg” is an interestingobject that may be hidden in a scene. If a viewer chooses, the viewercan pause the video at a scene, step into the scene, and look or searchfor “Easter Egg” that may be hidden or at least not obvious in thepre-rendered scene using the RVE system 100.

Example RVE Network Environments

Embodiments of real-time video explorer (RVE) systems that implement oneor more of the various methods as described herein, for example anexample RVE system 100 as illustrated in FIG. 1B, 6B, or 6C thatimplements the methods as illustrated in FIGS. 2 through 5, may beimplemented in the context of a service provider that providesvirtualized resources (e.g., virtualized computing resources,virtualized storage resources, virtualized database (DB) resources,etc.) on a provider network to clients of the service provider, forexample as illustrated in FIG. 13. Virtualized resource instances on theprovider network 2500 may be provisioned via one or more providernetwork services 2502 and may be rented or leased to clients of theservice provider, for example to an RVE system provider 2590 thatimplements RVE system 2510 on provider network 2502. At least some ofthe resource instances on the provider network 2500 may be computingresources 2522 implemented according to hardware virtualizationtechnology that enables multiple operating systems to run concurrentlyon a host computer, i.e. as virtual machines (VMs) on the host. Otherresource instances (e.g., storage resources 2552) may be implementedaccording to one or more storage virtualization technologies thatprovide flexible storage capacity of various types or classes of storageto clients of the provider network. Other resource instances (e.g.,database (DB) resources 2554) may be implemented according to othertechnologies.

In at least some embodiments, the provider network 2500, via theservices 2502, may enable the provisioning of logically isolatedsections of the provider network 2500 to particular clients of theservice provider as client private networks on the provider network2500. At least some of a client's resources instances on the providernetwork 2500 may be provisioned in the client's private network. Forexample, in FIG. 13, RVE system 2510 may be implemented as or in aprivate network implementation of RVE system provider 2590 that isprovisioned on provider network 2500 via one or more of the services2502.

The provider network 2500, via services 2502, may provide flexibleprovisioning of resource instances to clients in which virtualizedcomputing and/or storage resource instances or capacity can beautomatically added to or removed from a client's configuration on theprovider network 2500 in response to changes in demand or usage, thusenabling a client's implementation on the provider network 2500 toautomatically scale to handle computation and/or data storage needs. Forexample, one or more additional computing resources 2522A, 2522B, 2522C,and/or 2522D may be automatically added to RVE system 2510 in responseto an increase in the number of RVE clients 2582 accessing RVE system2510 to play back and explore video content as described herein. If andwhen usage drops below a threshold, computing and data storage resourcesthat are no longer necessary can be removed.

In at least some embodiments, RVE system provider 2590 may access one ormore of services 2502 of the provider network 2500 via applicationprogramming interfaces (APIs) to the services 2502 to configure andmanage an RVE system 2510 on the provider network 2500, the RVE system2510 including multiple virtualized resource instances (e.g., computingresources 2522, storage resources 2552, DB resources 2554, etc.).

Provider network services 2502 may include but are not limited to, oneor more hardware virtualization services for provisioning computingresource 2522, one or more storage virtualization services forprovisioning storage resources 2552, and one or more database (DB)services for provisioning DB resources 2554. In some implementations,RVE system provider 2590 may access two or more of these providernetwork services 2502 via respective APIs to provision and managerespective resource instances in RVE system 2510. However, in someimplementations, RVE system provider 2590 may instead access a singleservice (e.g., a streaming service 2504) via an API to the service 2504;this service 2504 may then interact with one or more other providernetwork services 2502 on behalf of the RVE system provider 2590 toprovision the various resource instances in the RVE system 2510.

In some embodiments, provider network services 2502 may include astreaming service 2504 for creating, deploying, and managing datastreaming applications such as an RVE system 2510 on a provider network2500. Many consumer devices, such as personal computers, tables, andmobile phones, have hardware and/or software limitations that limit thedevices' capabilities to perform 3D graphics processing and rendering ofvideo data in real time. In at least some embodiments, a streamingservice 2504 may be used to implement, configure, and manage an RVEsystem 2510 that leverages computation and other resources of theprovider network 2500 to enable real-time, low-latency 3D graphicsprocessing and rendering of video on provider network 2500, and thatimplements a streaming service interface 2520 (e.g., an applicationprogramming interface (API)) for receiving RVE client 2582 input and forstreaming video content including real-time rendered video as well aspre-recorded video to respective RVE clients 2582. In at least someembodiments, the streaming service 2504 may manage, for RVE systemprovider 2590, the deployment, scaling, load balancing, monitoring,version management, and fault detection and recovery of the server-sideRVE system 2510 logic, modules, components, and resource instances. Viathe streaming service 2504, the RVE system 2510 can be dynamicallyscaled to handle computational and storage needs, regardless of thetypes and capabilities of the devices that the RVE clients 2582 areimplemented on.

In at least some embodiments, at least some of the RVE clients 2582 mayimplement an RVE client interface 2684 as shown in FIG. 14 forcommunicating user input and interactions to RVE system 2510 accordingto the streaming service interface 2520, and for receiving andprocessing video streams and other content received from the streamingservice interface 2520. In at least some embodiments, the streamingservice 2504 may also be leveraged by the RVE system provider 2590 todevelop and build RVE clients 2582 for various operating system (OS)platforms on various types of client devices (e.g., tablets,smartphones, desktop/notebook computers, etc.).

Referring to FIG. 13, in at least some embodiments, data including butnot limited to video content may be streamed from the streaming serviceinterface 2520 to the RVE client 2582 according to a streaming protocol.In at least some embodiments, data including but not limited to userinput and interaction may be sent to the streaming service interface2520 from the RVE client 2582 according to the streaming protocol. In atleast some embodiments, the streaming service interface 2520 may receivevideo content (e.g., rendered video frames) from a video playback module(not shown) and/or from a rendering 2560 module, package the videocontent according to the streaming protocol, and stream the videoaccording to the protocol to respective RVE client(s) 2582 viaintermediate network 2570. In at least some embodiments, an RVE clientinterface 2684 of the RVE client 2582 may receive a video stream fromthe streaming service interface 2520, extract the video content from thestreaming protocol, and forward the video to a display component of therespective client device for display.

Referring to FIG. 13, an RVE system provider 2590 may develop and deployan RVE system 2510, leveraging one or more of services 2502 to configureand provision RVE system 2510. As shown in FIG. 13, the RVE system 2510may include and may be implemented as multiple functional modules orcomponents, with each module or component including one or more providernetwork resources. In this example, RVE system 2510 includes a streamingservice interface 2520 component that includes computing resources2522A, an RVE control module 2530 that includes computing resources2522B, 3D graphics processing 2540 module that includes computingresources 2522C, 3D graphics rendering 2560 module that includescomputing resources 2522D, and data storage 2550 that includes storageresources 2552 and database (DB) resources 2554. Note that an RVE system2510 may include more or fewer components or modules, and that a givenmodule or component may be subdivided into two or more submodules orsubcomponents. Also note that two or more of the modules or componentsas shown can be combined; for example, 3D graphics processing 2540module and 3D graphics rendering 2560 module may be combined to form acombined 3D graphics processing and rendering 108 module as shown inFIG. 1B.

One or more computing resources 2522 may be provisioned and configuredto implement the various modules or components of the RVE system 2510.For example streaming service interface 2520, RVE control module 2530,3D graphics processing 2540 module, and 3D graphics rendering 2560 mayeach be implemented as or on one or more computing resources 2522. Insome embodiments, two or more computing resources 2522 may be configuredto implement a given module or component. For example, two or morevirtual machine instances may implement an RVE control module 2530.However, in some embodiments, an instance of a given module (e.g., aninstance of 3D graphics processing 2540 module, or an instance of 3Dgraphics rendering 2560 module) may be implemented as or on each of thecomputing resource 2522 instances shown in the module. For example, insome implementations, each computing resource 2522 instance may be avirtual machine instance that is spun up from a machine imageimplementing a particular module, for example a 3D graphics processing2540 module, that is stored on storage resource(s) 2552.

In at least some embodiments, computing resources 2522 may bespecifically provisioned or configured to support particular functionalcomponents or modules of the RVE system 2510. For example, computingresources 2522C of 3D graphics processing 2540 module and/or computingresources 2522D of 3D graphics rendering module 2560 may be implementedon devices that include hardware support for 3D graphics functions, forexample graphics processing units (GPUs). As another example, thecomputing resources 2522 in a given module may be fronted by a loadbalancer provisioned through a provider network service 2502 thatperforms load balancing across multiple computing resource instances2522 in the module.

In at least some embodiments, different ones of computing resources 2522of a given module may be configured to perform different functionalitiesof the module. For example, different computing resources 2522C of 3Dgraphics processing 2540 module and/or different computing resources2522D of 3D graphics rendering module 2560 may be configured to performdifferent 3D graphics processing functions or apply different 3Dgraphics techniques. In at least some embodiments, different ones of thecomputing resources 2522 of 3D graphics processing 2540 module and/or 3Dgraphics rendering module 2560 may be configured with different 3Dgraphics applications. As an example of using different 3D graphicsprocessing functions, techniques, or applications, when renderingobjects for video content to be displayed, 3D data for the object may beobtained that needs to be processed according to specific functions,techniques, or applications to generate a 3D model of the object and/orto render a 2D representation of the object for display.

Storage resources 2552 and/or DB resources 2554 may be configured andprovisioned for storing, accessing, and managing RVE data including butnot limited to: pre-recorded video and new video content generated usingRVE system 2510; 3D data and 3D object models, and other 3D graphicsdata such as textures, surfaces, and effects; user information andclient device information; and information and data related to videosand video content such as information about particular objects. As notedabove, storage resources 2552 may also store machine images ofcomponents or modules of RVE system 2510. In at least some embodiments,RVE data including but not limited to video, 3D graphics data, objectdata, and user information may be accessed from and stored/provided toone or more sources or destinations eternal to RVE system 2510 onprovider network 2500 or external to provider network 2500.

Example Streaming Service Implementation

FIG. 14 illustrates an example network-based RVE system environment inwhich a streaming service 2504 is used to provide rendered video andsound to RVE clients, according to at least some embodiments. In atleast some embodiments, an RVE environment may include an RVE system2600 and one or more client devices 2680. The RVE system 2600 has accessto stores or other sources of pre-rendered, pre-recorded video, shown asvideo source(s) 2650. In at least some embodiments, the RVE system 100may also have access to stores or other sources of data and informationincluding but not limited to 3D graphics data and user information,shown as data source(s) 2660.

RVE system 2600 may include a front-end streaming service interface 2602(e.g., an application programming interface (API)) for receiving inputfrom RVE clients 2682 and streaming output to RVE clients 2682, andbackend data interface(s) 2603 for storing and retrieving data includingbut not limited to video, object, user, and other data and informationas described herein. The streaming service interface 2602 may, forexample, be implemented according to a streaming service 2504 asillustrated in FIG. 13. RVE system 2600 may also include video playbackand recording 2606 module(s), 3D graphics processing and rendering 2608module(s), and RVE control module 2604.

In response to user selection of a video for playback, video playbackand recording 2606 module(s) may obtain pre-rendered, pre-recorded videofrom a video source 2650, process the video as necessary, and stream thepre-recorded video to the respective client device 2680 via streamingservice interface 2602. During an RVE event in which the user pauses avideo being played back, steps into a scene, and explores and possiblymodifies the scene, 3D graphics processing and rendering 2608 module mayobtain 3D data from one or more data sources 2660, generate a 3D modeledworld for the scene according to the 3D data, render 2D representationsof the 3D modeled world from user-controlled camera viewpoints, andstream the real-time rendered video to the respective client device 2680via streaming service interface 2602. In at least some embodiments, thenewly rendered video content can be recorded by video playback andrecording 2606 module(s).

The RVE system 100 may also include an RVE control module 2604 thatreceives input and interactions from an RVE client 2682 on a respectiveclient device 2680 via streaming service interface 2602, processes theinput and interactions, and directs operations of video playback andrecording 2606 module(s) and 3D graphics processing and rendering 2608module accordingly. In at least some embodiments, RVE control module2604 may also track operations of video playback and recording 2606module(s). For example, RVE control module 104 may track playback of agiven video through video playback and recording 2606 module(s). moduleso that RVE control module 2604 can determine which scene is currentlybeing played back to a given client device 180.

In at least some embodiments, RVE client 2682 may implement a streamingservice client interface as RVE client interface 2684. User interactionswith a video being played back to the client device 2680, for exampleusing RVE controls 188 as shown in FIG. 1C and as implemented on theclient device 2680, may be sent from client device 2680 to RVE system2600 according to the streaming service interfaces 2684 and 2602. Ratherthan performing rendering of new 3D content on the client device 2680,3D graphics processing and rendering 2608 module(s) of RVE system 2600may generate and render new video content for scenes being explored inreal-time in response to the user input received from RVE client 2680.Streaming service interface 2602 may stream video content from RVEsystem 2699 to RVE client 2682 according to a streaming protocol. At theclient device 2680, the RVE client interface 2685 receives the streamedvideo, extracts the video from the stream protocol, and provides thevideo to the RVE client 2682, which displays the video to the clientdevice 2680.

Example Provider Network Environment

Embodiments of real-time video exploration (RVE) systems as describedherein may be implemented in the context of a service provider thatprovides resources (e.g., computing resources, storage resources,database (DB) resources, etc.) on a provider network to clients of theservice provider. FIG. 15 illustrates an example service providernetwork environment in which embodiments of RVE systems may beimplemented. FIG. 15 schematically illustrates an example of a providernetwork 2910 that can provide computing and other resources to users2900 a and 2900 b (which may be referred herein singularly as user 2900or in the plural as users 2900) via user computers 2902 a and 2902 b(which may be referred herein singularly as computer 2902 or in theplural as computers 2902) via a intermediate network 2930. Providernetwork 2910 may be configured to provide the resources for executingapplications on a permanent or an as-needed basis. In at least someembodiments, resource instances may be provisioned via one or moreprovider network services 2911, and may be rented or leased to clientsof the service provider, for example to an RVE system provider 2970. Atleast some of the resource instances on the provider network 2910 (e.g.,computing resources) may be implemented according to hardwarevirtualization technology that enables multiple operating systems to runconcurrently on a host computer (e.g., a host 2916), i.e. as virtualmachines (VMs) 2918 on the host.

The computing resources provided by provider network 2910 may includevarious types of resources, such as gateway resources, load balancingresources, routing resources, networking resources, computing resources,volatile and non-volatile memory resources, content delivery resources,data processing resources, data storage resources, database resources,data communication resources, data streaming resources, and the like.Each type of computing resource may be general-purpose or may beavailable in a number of specific configurations. For example, dataprocessing resources may be available as virtual machine instances thatmay be configured to provide various services. In addition, combinationsof resources may be made available via a network and may be configuredas one or more services. The instances may be configured to executeapplications, including services such as application services, mediaservices, database services, processing services, gateway services,storage services, routing services, security services, encryptionservices, load balancing services, and so on. These services may beconfigurable with set or custom applications and may be configurable insize, execution, cost, latency, type, duration, accessibility, and inany other dimension. These services may be configured as availableinfrastructure for one or more clients and can include one or moreapplications configured as a platform or as software for one or moreclients.

These services may be made available via one or more communicationsprotocols. These communications protocols may include, for example,hypertext transfer protocol (HTTP) or non-HTTP protocols. Thesecommunications protocols may also include, for example, more reliabletransport layer protocols, such as transmission control protocol (TCP),and less reliable transport layer protocols, such as user datagramprotocol (UDP). Data storage resources may include file storage devices,block storage devices and the like.

Each type or configuration of computing resource may be available indifferent sizes, such as large resources consisting of many processors,large amounts of memory and/or large storage capacity, and smallresources consisting of fewer processors, smaller amounts of memoryand/or smaller storage capacity. Customers may choose to allocate anumber of small processing resources as web servers and/or one largeprocessing resource as a database server, for example.

Provider network 2910 may include hosts 2916 a and 2916 b (which may bereferred herein singularly as host 2916 or in the plural as hosts 2916)that provide computing resources. These resources may be available asbare metal resources or as virtual machine instances 2918 a-d (which maybe referred herein singularly as virtual machine instance 2918 or in theplural as virtual machine instances 2918). Virtual machine instances2918 c and 2918 d are shared state virtual machine (“SSVM”) instances.The SSVM virtual machine instances 2918 c and 2918 d may be configuredto perform all or any portion of the real-time video explorer (RVE)system and RVE methods as described herein. As should be appreciated,while the particular example illustrated in FIG. 15 includes one SSVM2918 virtual machine in each host, this is merely an example. A host2916 may include more than one SSVM 2918 virtual machine or may notinclude any SSVM 2918 virtual machines.

The availability of virtualization technologies for computing hardwarehas afforded benefits for providing large scale computing resources forcustomers and allowing computing resources to be efficiently andsecurely shared between multiple customers. For example, virtualizationtechnologies may allow a physical computing device to be shared amongmultiple users by providing each user with one or more virtual machineinstances hosted by the physical computing device. A virtual machineinstance may be a software emulation of a particular physical computingsystem that acts as a distinct logical computing system. Such a virtualmachine instance provides isolation among multiple operating systemssharing a given physical computing resource. Furthermore, somevirtualization technologies may provide virtual resources that span oneor more physical resources, such as a single virtual machine instancewith multiple virtual processors that span multiple distinct physicalcomputing systems.

Referring to FIG. 15, intermediate network 2930 may, for example, be apublicly accessible network of linked networks and possibly operated byvarious distinct parties, such as the Internet. In other embodiments,intermediate network 2930 may be a local and/or restricted network, suchas a corporate or university network that is wholly or partiallyinaccessible to non-privileged users. In still other embodiments,intermediate network 2930 may include one or more local networks withaccess to and/or from the Internet.

Intermediate network 2930 may provide access to one or more clientdevices 2902. User computers 2902 may be computing devices utilized byusers 2900 or other customers of provider network 2910. For instance,user computer 2902 a or 2902 b may be a server, a desktop or laptoppersonal computer, a tablet computer, a wireless telephone, a personaldigital assistant (PDA), an e-book reader, a game console, a set-top boxor any other computing device capable of accessing provider network 2910via wired and/or wireless communications and protocols. In someinstances, a user computer 2902 a or 2902 b may connect directly to theInternet (e.g., via a cable modem or a Digital Subscriber Line (DSL)).Although only two user computers 2902 a and 2902 b are depicted, itshould be appreciated that there may be multiple user computers.

User computers 2902 may also be utilized to configure aspects of thecomputing, storage, and other resources provided by provider network2910 via provider network services 2911. In this regard, providernetwork 2910 might provide a gateway or web interface through whichaspects of its operation may be configured through the use of a webbrowser application program executing on a user computer 2902.Alternatively, a stand-alone application program executing on a usercomputer 2902 might access an application programming interface (API)exposed by a service 2911 of provider network 2910 for performing theconfiguration operations. Other mechanisms for configuring the operationof various resources available at provider network 2910 might also beutilized.

Hosts 2916 shown in FIG. 15 may be standard host devices configuredappropriately for providing the computing resources described above andmay provide computing resources for executing one or more servicesand/or applications. In one embodiment, the computing resources may bevirtual machine instances 2918. In the example of virtual machineinstances, each of the hosts 2916 may be configured to execute aninstance manager 2920 a or 2920 b (which may be referred hereinsingularly as instance manager 2920 or in the plural as instancemanagers 2920) capable of executing the virtual machine instances 2918.An instance manager 2920 may be a hypervisor or virtual machine monitor(VMM) or another type of program configured to enable the execution ofvirtual machine instances 2918 on a host 2916, for example. As discussedabove, each of the virtual machine instances 2918 may be configured toexecute all or a portion of an application or service.

In the example provider network 2910 shown in FIG. 15, a router 2914 maybe utilized to interconnect the hosts 2916 a and 2916 b. Router 2914 mayalso be connected to gateway 2940, which is connected to intermediatenetwork 2930. Router 2914 may be connected to one or more loadbalancers, and alone or in combination may manage communications withinprovider network 2910, for example, by forwarding packets or other datacommunications as appropriate based on characteristics of suchcommunications (e.g., header information including source and/ordestination addresses, protocol identifiers, size, processingrequirements, etc.) and/or the characteristics of the network (e.g.,routes based on network topology, subnetworks or partitions, etc.). Itwill be appreciated that, for the sake of simplicity, various aspects ofthe computing systems and other devices of this example are illustratedwithout showing certain conventional details. Additional computingsystems and other devices may be interconnected in other embodiments andmay be interconnected in different ways.

In the example provider network 2910 shown in FIG. 15, a host manager2915 may also be employed to at least in part direct variouscommunications to, from and/or between hosts 2916 a and 2916 b. WhileFIG. 15 depicts router 2914 positioned between gateway 2940 and hostmanager 2915, this is given as an example configuration and is notintended to be limiting. In some cases, for example, host manager 2915may be positioned between gateway 2940 and router 2914. Host manager2915 may, in some cases, examine portions of incoming communicationsfrom user computers 2902 to determine one or more appropriate hosts 2916to receive and/or process the incoming communications. Host manager 2915may determine appropriate hosts to receive and/or process the incomingcommunications based on factors such as an identity, location or otherattributes associated with user computers 2902, a nature of a task withwhich the communications are associated, a priority of a task with whichthe communications are associated, a duration of a task with which thecommunications are associated, a size and/or estimated resource usage ofa task with which the communications are associated and many otherfactors. Host manager 2915 may, for example, collect or otherwise haveaccess to state information and other information associated withvarious tasks in order to, for example, assist in managingcommunications and other operations associated with such tasks.

It should be appreciated that the network topology illustrated in FIG.15 has been greatly simplified and that many more networks andnetworking devices may be utilized to interconnect the various computingsystems disclosed herein. These network topologies and devices should beapparent to those skilled in the art.

It should also be appreciated that provider network 2910 described inFIG. 15 is given by way of example and that other implementations mightbe utilized. Additionally, it should be appreciated that thefunctionality disclosed herein might be implemented in software,hardware or a combination of software and hardware. Otherimplementations should be apparent to those skilled in the art. Itshould also be appreciated that a host, server, gateway or othercomputing device may comprise any combination of hardware or softwarethat can interact and perform the described types of functionality,including without limitation desktop or other computers, databaseservers, network storage devices and other network devices, PDAs,tablets, cell phones, wireless phones, pagers, electronic organizers,Internet appliances, television-based systems (e.g., using set top boxesand/or personal/digital video recorders), game systems and gamecontrollers, and various other consumer products that includeappropriate communication and processing capabilities. In addition, thefunctionality provided by the illustrated modules may in someembodiments be combined in fewer modules or distributed in additionalmodules. Similarly, in some embodiments the functionality of some of theillustrated modules may not be provided and/or other additionalfunctionality may be available.

Illustrative System

In at least some embodiments, a computing device that implements aportion or all of the technologies as described herein may include ageneral-purpose computer system that includes or is configured to accessone or more computer-readable media, such as computer system 3000illustrated in FIG. 16. In the illustrated embodiment, computer system3000 includes one or more processors 3010 coupled to a system memory3020 via an input/output (I/O) interface 3030. Computer system 3000further includes a network interface 3040 coupled to I/O interface 3030.

In various embodiments, computer system 3000 may be a uniprocessorsystem including one processor 3010, or a multiprocessor systemincluding several processors 3010 (e.g., two, four, eight, or anothersuitable number). Processors 3010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 3010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 3010 may commonly,but not necessarily, implement the same ISA.

System memory 3020 may be configured to store instructions and dataaccessible by processor(s) 3010. In various embodiments, system memory3020 may be implemented using any suitable memory technology, such asstatic random access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above, are shown stored within system memory 3020 as code 3025and data 3026.

In one embodiment, I/O interface 3030 may be configured to coordinateI/O traffic between processor 3010, system memory 3020, and anyperipheral devices in the device, including network interface 3040 orother peripheral interfaces. In some embodiments, I/O interface 3030 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 3020) intoa format suitable for use by another component (e.g., processor 3010).In some embodiments, I/O interface 3030 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 3030 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 3030, such as an interface to system memory 3020, may beincorporated directly into processor 3010.

Network interface 3040 may be configured to allow data to be exchangedbetween computer system 3000 and other devices 3060 attached to anetwork or networks 3050, such as other computer systems or devices, forexample. In various embodiments, network interface 3040 may supportcommunication via any suitable wired or wireless general data networks,such as types of Ethernet network, for example. Additionally, networkinterface 3040 may support communication viatelecommunications/telephony networks such as analog voice networks ordigital fiber communications networks, via storage area networks such asFibre Channel SANs, or via any other suitable type of network and/orprotocol.

In some embodiments, system memory 3020 may be one embodiment of acomputer-readable medium configured to store program instructions anddata as described above for implementing embodiments of thecorresponding methods and apparatus. However, in other embodiments,program instructions and/or data may be received, sent or stored upondifferent types of computer-readable media. Generally speaking, acomputer-readable medium may include non-transitory storage media ormemory media such as magnetic or optical media, e.g., disk or DVD/CDcoupled to computer system 3000 via I/O interface 3030. A non-transitorycomputer-readable storage medium may also include any volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR SDRAM, RDRAM, SRAM,etc.), ROM, etc, that may be included in some embodiments of computersystem 3000 as system memory 3020 or another type of memory. Further, acomputer-readable medium may include transmission media or signals suchas electrical, electromagnetic, or digital signals, conveyed via acommunication medium such as a network and/or a wireless link, such asmay be implemented via network interface 3040.

Conclusion

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-readable medium. Generally speaking, acomputer-readable medium may include storage media or memory media suchas magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc, as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent example embodiments of methods. The methods may be implementedin software, hardware, or a combination thereof. The order of method maybe changed, and various elements may be added, reordered, combined,omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A system, comprising: one or more computingdevices comprising one or more hardware processors and memory andconfigured to implement a real-time video exploration (RVE) systemcomprising: a playback module implemented via the one or more hardwareprocessors and memory and configured to begin playback of at least aportion of a pre-recorded video to a client device; and a graphicsprocessing and rendering module implemented via the one or more hardwareprocessors and memory and configured to: receive identification inputfrom the client device after playback of the pre-recorded video hasbegun, wherein the identification input identifies a particular objectof a plurality of objects contained in a scene of the pre-recordedvideo; pause playback of the scene in response to the identificationinput; receive manipulation input from the client device after playbackof the pre-recorded video has begun directing manipulation of theparticular object in the scene of the pre-recorded video; obtain a modelof the particular object according to graphics data for the pre-recordedvideo; manipulate the model of the particular object in a model of thescene according to the manipulation input; render new video of the sceneincluding a rendering of the manipulated model of the particular objectas manipulated by the manipulation input, wherein the particular objectin the scene is replaced with the manipulated model of the particularobject; stream the new video of the scene including the rendering of themanipulated model of the particular object to the client device whileplayback of the scene is paused; and resume playback of the pre-recordedvideo to the client device in response to resume input from the clientdevice, wherein streaming the new video is stopped in response to theresume input.
 2. The system as recited in claim 1, wherein themanipulation input directs repositioning of the particular object withinthe scene, and wherein the new video shows the particular objectrepositioned within the scene as directed by the manipulation input. 3.The system as recited in claim 1, wherein the manipulation input directsmovement of the particular object within the scene, and wherein the newvideo shows the particular object moving within the scene as directed bythe manipulation input.
 4. The system as recited in claim 1, wherein themanipulation input directs changing one or more of a viewing angle orposition relative to the particular object within the scene, and whereinthe new video shows the particular object from the changed viewing angleor position.
 5. The system as recited in claim 1, wherein themanipulation input directs manipulation of a component of the particularobject, and wherein the new video shows the particular object with thecomponent as manipulated as directed by the manipulation input.
 6. Thesystem as recited in claim 1, wherein the new video shows additionaldetail of the particular object that is not visible in the pre-recordedvideo.
 7. The system as recited in claim 1, wherein the manipulationinput interacts with an interface of the particular object as displayedon the client device, and wherein the new video shows a response to theinteraction with the interface of the particular object.
 8. The systemas recited in claim 1, wherein the one or more computing devices thatimplement the RVE system are on a provider network, and wherein thegraphics processing and rendering module is configured to leverage oneor more computing resources of the provider network to perform saidrendering in real time in response to the manipulation input.
 9. Thesystem as recited in claim 1, wherein the identification and themanipulation input are received from the client device according to anapplication programming interface (API) of the RVE system.
 10. Thesystem as recited in claim 1, wherein the particular object comprises athree-dimensional (3D) object, wherein the graphics data comprises 3Dgraphics data, and wherein the model of the particular object comprisesa 3D model.
 11. A method, comprising: performing, by a real-time videoexploration (RVE) system implemented on one or more computing devices:receiving identification input that identifies a particular object of aplurality of objects in a scene of a pre-recorded video, wherein theidentification input is received after playback of the pre-recordedvideo has begun; pausing playback of the scene in response to theidentification input; receiving manipulation input directingmanipulation of the particular object in the scene of the pre-recordedvideo, wherein the manipulation input is received after playback of thepre-recorded video has begun; obtaining a model of the particular objectaccording to graphics data for the pre-recorded video; manipulating themodel of the particular object in a model of the scene according to themanipulation input; rendering new video of the scene including arendering of the manipulated model of the particular object asmanipulated by the manipulation input, wherein the particular object inthe scene is replaced with the manipulated model of the particularobject; streaming the new video of the scene including the rendering ofthe manipulated model of the particular object to a client device whileplayback of the scene is paused; and resuming playback of thepre-recorded video to the client device in response to resume input fromthe client device, wherein streaming the new video is stopped inresponse to the resume input.
 12. The method as recited in claim 11,wherein the manipulation input directs repositioning the particularobject within the scene, and wherein said rendering new video of thescene including a rendering of the model of the particular object asmanipulated comprises rendering the particular object as repositionedwithin the scene.
 13. The method as recited in claim 11, wherein themanipulation input directs changing one or more of a viewing angle orposition relative to the particular object within the scene, and whereinsaid rendering new video of the scene including a rendering of the modelof the particular object as manipulated comprises rendering theparticular object from the changed viewing angle or position.
 14. Themethod as recited in claim 11, wherein the graphics processing andrendering module is further configured to obtain the model of theparticular object and manipulate the model of the particular objectwhile the pre-recorded video is paused.
 15. The method as recited inclaim 14, further comprising: in response to receiving theidentification input, requesting information corresponding to theparticular object; and receiving the information from a database,wherein the new video includes the received information.
 16. The methodas recited in claim 11, wherein the particular object comprises athree-dimensional (3D) object, wherein the graphics data comprises 3Dgraphics data, and wherein the model of the particular object comprisesa 3D model.
 17. A non-transitory computer-readable storage mediumstoring program instructions that when executed on one or more computerscause the one or more computers to implement a real-time videoexploration (RVE) system configured to: receive identification inputfrom a client device identifying a particular object of a plurality ofobjects in a scene of a pre-recorded video, wherein the identificationinput is received after playback of the pre-recorded video has begun;pause playback of the scene in response to the identification input;receive manipulation input from the client device directing manipulationof the particular object in the scene of the pre-recorded video; obtaina model of the particular object according to graphics data for thepre-recorded video; manipulate the model of the particular object in amodel of the scene according to the manipulation input; render new videoof the scene including a rendering of the manipulated model of theparticular object as manipulated by the manipulation input, wherein theparticular object in the scene is replaced with the manipulated model ofthe particular object; stream the new video of the scene including therendering of the manipulated model of the particular object to theclient device while playback of the scene is paused; and resume playbackof the pre-recorded video to the client device in response to resumeinput from the client device, wherein streaming the new video is stoppedin response to the resume input.
 18. The non-transitorycomputer-readable storage medium as recited in claim 17, wherein themanipulation input directs repositioning or moving the particular objectwithin the scene, and wherein the new video shows the particular objectas repositioned or moved within the scene.
 19. The non-transitorycomputer-readable storage medium as recited in claim 17, wherein themanipulation input directs manipulation of a component or interface ofthe particular object, and wherein the new video shows the particularobject with the component or interface as manipulated.
 20. Thenon-transitory computer-readable storage medium as recited in claim 17,wherein the particular object comprises a three-dimensional (3D) object,wherein the graphics data comprises 3D graphics data, and wherein themodel of the particular object comprises a 3D model.